text:
How does Bitcoin Core overcome the condition that chain work in pre-sync must be within one day of the current active chain tip when it is genesis?
Two conditions for the chain we got in the header pre-sync are:

to have minimal chain work
to be within 1 day worth of work from the active chain tip's work

When we start Bitcoin core node for the first time, the active chain tip is on genesis block. The chain we get in pre-sync has an extremely large amount of cumulative work. So, it will pass the first condition, but how will it pass the second condition? It will be much more than within 1 day worth of genesis block work.

It has to have at least as much work as within a day's worth of the active chain tip's work. More work is always better.

@PieterWuille My bad than


text:
What are all the consensus rules for the block version value?
I am interested in what are the all consensus rules for block version number in a block header?
Found some answer here.
Andrew Chow said that it can be whatever value we want except some specific values. What are these values?
Pieter Wuille in the same answer (comment section) gave a note that the version must be equal to/larger than 4 according to the BIP16, BIP65, BIP66. Thus, I would say that the version number of the block can't be 0, 1, 2, 3 and can be everything above it? Are these the specific values that Andrew was talking about? So, according to the consensus rules, the version number can be anything higher than 3? He also mentioned some rules from BIP 9, but as I understand, they are not consensus rules so they do not have to be respected.
Murch in his question/answer was also talking about the need for block version to be above 3 (>= 4) and about the fact that we must take care not to set the "wrong" bit (sign bit; top most bit) to 1. He also wrote that the top 3 bits must be 001 (according to BIP 9) to activate some "signaling for soft fork" rules from that BIP. However, it is not a consensus rule (as Pieter said), so we can ignore it and set these top 3 bits to 001 without any desire for signaling.
So for now I would say that the set of possible values is anything greater than 3 and that the most top bit (00000000 00000000 00000000 x0000000) can't be 1. Any other value can be set for the version, and even, I would say, 001 for the most top 3 bits can be set because in that case we would signal something random with other bits (but who cares).
So my question is, what are all the consensus rules for the block version field in the block header?

Hi Mudja, it seems to me that your question is answered in What restrictions does the version field in the block header have?, which you link. Could you please explain how your question differs?

@Murch Hi Murch. Yes, I saw that answer. I included it in my question (with some other answers).

@Murch Well, the answers given by you, Pieter and Andrew match partially. They didn't mention sign bit nor talk about 001 for BIP9. Also, you wrote "I'm aware of the following restrictions" which does not guarantee me that you wrote all restrictions. Also, Poseidon and you talk about all values greater than 3 where the most top 3 bits are not 001 are valid. However, I believe even if they are set to 001 and with the desire NOT to signal anything, it is still allowed by consensus. And finally, I want to see if I understood correctly what all of you wrote.

It seems to me that my answer clearly states that the only consensus rule is that the version has to be greater than three, so I don’t understand how my answer doesn’t answer this new question here.

@Murch Okay, so we can set any value greater than 3 where we can also set the three most significant bits to 001 (even if we do not signal anything, but who caress its not consensus rule). The only additional restriction is to NOT set the most significant bit (sign bit) to 1 because by that we will get negative numbers and automatically it is therefore less than 4. Did I understand correctly?

Yes, the version has to be greater than three.

@Murch Then everything is clear. Thanks for the help!

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Does hash_count field exists in getheaders/getblocks messages?
Ava Chow said it does not exist (link). If hash_count does not exist, how will the receiving peer know how many locators are in the message?
I am not sure if he maybe thought that hash_count does not exist for hash_stop or it does not even exist for locators.
Bitcoin wiki says it exists for both getheaders and getblocks:

https://en.bitcoin.it/wiki/Protocol_documentation#getheaders
https://en.bitcoin.it/wiki/Protocol_documentation#getblocks

Bitcoin for developers also mentions hash_count field:

https://developer.bitcoin.org/reference/p2p_networking.html#getblocks

If hash_count is no longer used, when did that change in bitcoin core happen and how was compatibility with old nodes maintained?

This is a matter of terminology, but all these descriptions are equivalent.
The locator is a vector of hashes. In Bitcoin's serialization framework, vectors are always serialized by first writing the number of objects, and then the serialization of those objects themselves, concatenated. This applies in many places:

The block hashes in a locator
The list of inventory objects in inv and getdata
The list of addresses in addr and addrv2
Transactions in a block
Transaction inputs in a transaction
Transaction outputs in a transaction
Bytes in a script
Number of witness items in a witness stack
...

Some descriptions of the protocol treat this count as a separate "field" before the object, while others treat it as part of the object itself. To someone familiar with the Bitcoin Core source code, the latter feels probably more familiar, as there is no actual variable "hash_count" in the implementation; it's just the length of the locator object that goes in and comes out.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
What happens when hash_stop in getblocks/getheaders messages has a wrong/nonexistent hash?
What happens when hash_stop contains a hash that the node that receives the getblocks/getheaders message doesn't have, or does, but that hash (that is, the block whose hash it is) is somewhere earlier in the chain (behind the hash in block locator)? Is the maximum number of elements (childs) then returned, which is 500 for getblocks and 2000 for getheaders?

getheaders only checks that hash_stop is part of the main chain if locator is empty. If it is, then it returns just that particular block header.
getblocks does not do any validation on hash_stop.
Both getheaders and getblocks will compare the block hashes exactly, so it only stops when it either reaches the limit, or sees a block hash that exactly matches hash_stop. If the hash_stop is not in the main chain after locator, then the maximum number of items will be returned.

From your first paragraph, it looks like getheaders, by setting hash_count to 0 and then providing hash (which represents hash_stop) is a way to obtain just one concrete header of some block? Before we could only get the whole block.

Yes, if you provided just hash_stop then you would get just that header. There is no hash_count.

getheaders/getblocks messages, according to bitcoin wiki, includes hash_count (https://en.bitcoin.it/wiki/Protocol_documentation#getheaders). Bitcoin for developers also mentions hash_count (https://developer.bitcoin.org/reference/p2p_networking.html#getblocks)

Ah, within the code, there is no separate hash_count. It's just part of the serialization for a block locator object.

Thanks. For future readers: https://bitcoin.stackexchange.com/a/121090/146389


text:
Meaning of xPub in SegWit wallet on blockchain.com and their legacy-type addresses
According to my understanding, an xPub of a Legacy sub-wallet should begin with xpub... and of a SegWit sub-wallet with zpub.... And the addresses begin with 1 and bc1 respectively.
But on blockchain.com (where I only have one wallet) for the SegWit sub-wallet I get an xPub beginning with xpub... instead of ypub.... And the addresses generated with this also begin with 1.
When exporting the xPub from Electrum native-SegWit sub-wallet with the same xPrv, I get an xPub beginning with zpub..., as expected.
Is there a purpose on providing the xPub for a SegWit sub-wallet instead of the zPub? Addresses created with the xPub don't show up neither on blockchain.com nor in Electrum.
After being curious about this, I used both xPubs on blockchain.com xpub explorer (https://www.blockchain.com/de/explorer/assets/btc/xpub/xpub6CM...) and found all transactions on the corresponding addresses. As expected, older ones on Legacy addresses, newer ones on SegWit addresses.
And to my surprise, with the xPub of the SegWit sub-wallet I also found an address with balance on it within the addresses beginning with 1. This address doesn't show up neither on blockchain.com nor in Electrum(Legacy & native-SegWig). But it is also found with this tool (https://blockpath.com/wallets/local/101?action=appxpub) with index m/0/2.
In addition to my first question, how is it possible to get access to this address?

To answer your first question: The format of the extended public key defines what kind of addresses are created from it. In other words xPub... and zPub... are both versions of the same key but zPub... additionally instructs the wallet to derive a segwit address from it. But blockchain.com does not use this standard and instead encodes it with xPub... standard.
You can use this open source tool to convert your extended public key to the other versions https://jlopp.github.io/xpub-converter/ if you want to see that the yPub... in Electrum matches your segwit xPub...
And to answer your second question: You can try to access the mystery address of your blockchain.com segwit wallet by using extended private key (xprv) of the segwit wallet and setting the derivation path to m/0'/0'/n' or m/44'/0'/n'.


text:
BOLT12: is there a way to store a detailed itemized receipt in any field?
When using BOLT12 offers in a point of sale application, is there any way to embed the entire detailed itemized receipt into the offer and invoice?
For example

4L of milk (400 sat)
2kg ground beef (600 sat)
1kg sugar (300 sat)
(1) tube toothpaste (150 sat)
(4) rolls toilet paper (250 sat)
etc.

Can we store this much data in offers and invoices and if so, what field?
Possibly we don't want to store this data in an offer because the offer should be short so it can be encoded in a small QR code, but is there any reason to keep it out of the invoice if that is sent through onion messaging? Is there any limit to how much data we can relay through onion messaging (seems like there is not an incentive based approach to onion messaging in lightning, so I'm not sure how nodes decide to relay messages for free)?


text:
BOLT12 merchant-pays-user flow: how does a merchant know the `invoice_node_id` of the person it received the `invoice` from in advance?
BOLT12 has a merchant-pays-user flow (https://github.com/lightning/bolts/blob/247e83d528a2a380e533e89f31918d7b0ce6a0c1/12-offer-encoding.md?plain=1#L54). In this, the merchant receives an invoice (in response to its invoice_request) and confirms the invoice_node_id to ensure it's about to pay the correct person, and makes a payment to the invoice (https://github.com/lightning/bolts/blob/247e83d528a2a380e533e89f31918d7b0ce6a0c1/12-offer-encoding.md?plain=1#L58).
How does a merchant know the invoice_node_id of the person if received the invoice from in advance, especially if the invoice_request was presented via e-mail, NFC, QR, etc. and not from an offer? Seems like the signature field from the invoice_request needs to be copied over to the invoice and verify that way.
OR, is this workflow meant for a refund only (and not a general purpose payout) and there is an original invoice that the refund workflow is referring to? If so, it still seems like there should be a signature involved and not rely on an invoice_node_id.


text:
broadcast ps2wsh transaction failed
Greeting,
I hope someone can guide me about what is wrong with my p2wsh transaction hex below
020000000001014cf3b8d35c7b67fa3b669900f2a42b2e8b8a90487c2fca2b913b4e9d24a7b94e5d00000000ffffffff01e80300000000000022002053461e7b7bebf5089234af05cc4bfa5447ec7a9bc611c1f8d47dfc570a0339430221876bf128fa0fe05d617494045dc8dc27a530ec4d9c80e8d35eac377bd9c4e5090024a821876bf128fa0fe05d617494045dc8dc27a530ec4d9c80e8d35eac377bd9c4e509008700000000

i tried decode the hex and got result below :
{
"addresses": [
    "tb1qlwfxkadgkjykp24pcs699x3yx7egvm26khcpjheysknwv0ahh4zqmnc7xe",
    "tb1q2drpu7mma06s3y354uzucjl623r7c75mccgur7x50h79wzsr89psmcywrq"
],
"block_height": -1,
"block_index": -1,
"confirmations": 0,
"double_spend": false,
"fees": 1000,
"hash": "6ad4fce6043d8012105d6b2da5da3d8519124d8bf8c97fa5c1efad27a3dd72d9",
"inputs": [
    {
        "addresses": [
            "tb1qlwfxkadgkjykp24pcs699x3yx7egvm26khcpjheysknwv0ahh4zqmnc7xe"
        ],
        "age": 2579745,
        "output_index": 93,
        "output_value": 2000,
        "prev_hash": "4eb9a7249d4e3b912bca2f7c48908a8b2e2ba4f20099663bfa677b5cd3b8f34c",
        "script_type": "pay-to-witness-script-hash",
        "sequence": 4294967295,
        "witness": [
            "876bf128fa0fe05d617494045dc8dc27a530ec4d9c80e8d35eac377bd9c4e50900",
            "a821876bf128fa0fe05d617494045dc8dc27a530ec4d9c80e8d35eac377bd9c4e5090087"
        ]
    }
],
"outputs": [
    {
        "addresses": [
            "tb1q2drpu7mma06s3y354uzucjl623r7c75mccgur7x50h79wzsr89psmcywrq"
        ],
        "script": "002053461e7b7bebf5089234af05cc4bfa5447ec7a9bc611c1f8d47dfc570a033943",
        "script_type": "pay-to-witness-script-hash",
        "value": 1000
    }
],
"preference": "low",
"received": "2024-02-28T06:29:26.761160172Z",
"relayed_by": "54.226.234.175",
"size": 168,
"total": 1000,
"ver": 2,
"vin_sz": 1,
"vout_sz": 1,
"vsize": 113}

When i tried broadcast the raw transaction hex above, i got error "non-mandatory-script-verify-flag (Script evaluated without error but finished with a false/empty top stack element)"
I'm new to bitcoin development, and still struggle to understand by example.

The witness script is the last element pushed on witness, and in your case is a821....
Your address is tb1qlwf...
A P2WSH address is sha256(witness script), let's verify that:
$ hal address -t create --script a821876bf128fa0fe05d617494045dc8dc27a530ec4d9c80e8d35eac377bd9c4e5090087

{
  "p2sh": "2NGYBv4ryXa55wg8wTfMUhUSEAqJhe8AB4j",
  "p2wsh": "tb1qlwfxkadgkjykp24pcs699x3yx7egvm26khcpjheysknwv0ahh4zqmnc7xe",
  "p2shwsh": "2NAx9vPRkc5obFLPYG98YnpRFo7TodYkLqx",
  "p2tr": "tb1prrmgv7lpzwqgt7vkkdfndgzpwmpz4lts79j8f8htsaz8p08fmv9sjvqhky"
}

Yes, the value for p2wsh (tb1qlwf...) matches the address. So far so good.
Now let's decode your witness script a821...:
$ hal script decode a821876bf128fa0fe05d617494045dc8dc27a530ec4d9c80e8d35eac377bd9c4e5090087

OP_SHA256 OP_PUSHBYTES_33 876bf128fa0fe05d617494045dc8dc27a530ec4d9c80e8d35eac377bd9c4e50900 OP_EQUAL

This means it expects some value, which sha256 hash will be equal to 876b....
The value your are passing is 876b..., i.e., the same value as the hash you're expecting. But this value, after being sha256 will be totally different, so the script will return false on OP_EQUAL
Let's verify that:
$ hal hash -t sha256 876bf128fa0fe05d617494045dc8dc27a530ec4d9c80e8d35eac377bd9c4e50900

80afc01a83b056ec8861420311387ab2dc57aab6b4b564c060b65049a8eda1de

Since 876b... and 80af... are different, the script fails.
To fix this, on the first element of the witness you need to pass the value which sha256 is equal to 876b...


text:
Partial private key recovery
I had the pleasure of setting up a Bitcoin wallet using the Blue Wallet app and diligently noted down the 52 characters of my private key on a piece of paper for backup purposes.
Upon revisiting the paper to import my private key into a Wallet App, I discovered, that it was not recognized by the system.
It appears I may have inadvertently transcribed 2-3 characters inaccurately.
Could you kindly advise if there's a method by which I might retrieve my private key through Python, assuming the misspelling of up to three characters..?

I apologize for the lack of clarity in my explanation. I imported a private key, which was generated externally, into the Blue Wallet and subsequently destroyed the written copy of that private key.
Afterward, when I attempted to transcribe the 52-character private key from the backup tab in Blue Wallet onto paper, it seems I may have incorrectly noted down 2-3 characters.
Therefore, my inquiry is not specifically about Blue Wallet. I'm asking if there's a way to recover the correct sequence when 2-3 characters out of an arbitrary 52-character string are missing, not involving seed phrases.

Hi! I checked the Blue Wallet app from Play Store. I can see only the seed words as an export option. Do you have a list of words or a random alphanumeric string? If you have the random string, where can you export it in the app? A standard private key (256-bit) should have 64 chars

Thank you for your reply. I believe my explanation might have been insufficient. I have added more details to the main text.

Yes, There is a way but it may not be 100% accurate.
52 character WIF means you have a compressed private key.
Which means the Base58 decoded byte representation of that key should have compressed byte flag 01 at exactly offset 33
WIF compressed layout:

mainnet
privatekey
compressed flag
checksum

1 byte
32 byte
1 byte
4 byte

80
variety
01
variety

Now since you don't know which characters are wrong, we can only assume only byte offset 0 is correct and to find position where you mistyped the key it's hard but you can narrow it down like this:
Decode WIF to hexadecimal.
Check if the hexadecimal representation has 01 at position 67-68.
If not the error is too hard to correct.
If yes then there is a chance but you have to analyze deeper.
I did this with someone's WIF few back and successfully recovered it but you need one of these or both:
Address
Public key


text:
How feasible is a double spend attack?
Each source of information I look at that tackles this problem states that this type of fraud is essentially impossible at best and highly infeasible at worst. By my (very limited) understanding, the success of this attack is actually down to timing and luck rather than it needing a 51% control of hashing compute. The scenario I've been thinking about:
Initial State and assumptions:

All nodes are at block height
No forks exist; all nodes are in consensus
Each new block takes the full normally distributed 10 minutes to confirm
In the below case, the 'fraudulent block' contains a transaction signed by the bad actor to themselves or to a colluding partner to consume the unspent input - to prevent the 'real' transaction confirming for the 'real' recipient. Yes - advise the 'real' recipient to wait for enough block confirmations to take place so that the attack is pointless - but lets assume someone is scammed into handing over goods/services/digital assets on the basis that the 'real' transaction is submitted and pending. I want to focus more on how the technology accounts for the possibility of the problem, rather than human intuition and intervention.

Fraudulent Block Submission:

An attacker successfully mines a fraudulent block +1 and propagates it through the network. This block contains the double spend attack transaction, leaving the 'real' transaction in the mempool where it's rejected because its considered a spent transaction, therefore excluded from any new block
Within a minute or two, the majority of the network receives and validates this block.
Nodes recognize +1 as the longest chain and switch to mining on top of it.

Mining Block +2:

Assume the next block +2 takes the full (normally distributed) 10 minutes to mine after +1 has been accepted.
Within the first 5 minutes, almost all nodes would have received, validated, and started working on block +1, given the propagation time is typically less than a couple of minutes.

Given these conditions, here's how the scenario unfolds:
Propagation of Block +1:

Once block +1 is propagated and validated across the network, all honest nodes will switch to the chain ending in +1.

Mining Block +2:

With all nodes working on the new longest chain, they will mine block +2 on top of +1.
The network will be in consensus that block +1 is part of the main chain, and thus any new blocks (including +2) will be built on top of +1.

In this scenario, assuming the fraudulent block +1 propagates and is accepted as the longest chain by the majority of the network, all subsequent blocks (like +2) will indeed be built on this chain. In Summary:

Propagation and Validation: Block +1 propagates and is validated quickly (within a couple of minutes), leading all nodes to switch to this chain.
Network Consensus: By 5 minutes into the 10-minute block interval, all nodes will be mining on the +1 chain due to its acceptance as the longest chain.
Subsequent Blocks: The next block +2 is highly likely to be mined on top of +1 since the entire network considers +1 as part of the longest chain.

So, if we assume that the above case holds true it seems to me at least that a double spend attack IS quite feasible under the right conditions, so what other protections are in place to

Prevent this happening in the first place
Unwind in the event that there are multiple block confirmations subsequently

Can you describe exactly what way is your fraudulent block fraudulent? For example does it contain a transaction created by the same person as created the "real transaction" which spends the same inputs but has different outputs?

Yes - this is the scenario exactly, i.e., payment is instead sent from the bad actor to themselves or to a colluding partner instead of the 'real' recipient. I will update the post - thanks

See updated assumption, 4

what other protections are in place to ... Prevent this happening in the first place ...*

None other than the one you identified. If a vendor of goods and services provides those before a payment transaction has been sufficiently confirmed, there is a risk that the payment will not occur. There is a longstanding convention that six confirmations provide a good level of certainty.

... Unwind in the event that there are multiple block confirmations subsequently

There is no provision for this for individual allegations of fraud. There is no Bitcoin body capable of determining which party is telling the truth. This is the same with other forms of cash. If you hand over your car for a handful of US Dollars in cash that turns out to be forgeries, you go to the police and courts, not to the Bureau of Engraving and Printing in the US Treasury department.

"fraudulent block"

That block contains thousands of valid honest transactions, so labelling it a "fraudulent block" seems a bit tendentious. Consider the old system of paper checks (cheques): After a vendor receives a check they take it to their bank who temporarily credit the vendors account pending clearance. The vendors bank send the check to the payer's bank, the payer's bank may reply saying the payer has insufficient funds. The vendors bank will then debit the vendors account the amount of the bounced check.
Describing the truck that delivers the check, along with thousands of others, as a "fraudulent truck" would be a bit strange and potentially misleading.

Bitcoin is cash, not a bank account.
Well-confirmed Bitcoin transactions are not reversible.
A payment system does not adjudicate disputes.

I am aligned that Bitcoin is cash, akin to real dollars in that there are no inherent protections in the exchange of cash itself. Fraudulent block is incorrect, yes. The problem refers to a fraudulent transaction by a bad actor within a valid block, but I'm not disputing the real world implication. I'm interested in learning if the protocol has other methods to verify integrity of unspent transactions. I don't like the explanation that this type of attack is impossible without 51% of compute, in reality its easier to achieve than that. You answer this in the first part of your response - None

@M.White: An unconfirmed transaction is a payment promise. A confirmed transaction is a payment.


text:
SIGHASH_ANYONECANPAY mediated tx modification
If Alice broadcasts a tx with an input tx_in1 and output tx_out1 signed with SIGHASH_ALL | SIGHASH_ANYONECANPAY, it lives temporarily in the mempool. Bob chooses to be a part of the crowdfunding and he adds an input tx_in2 to the transaction, but then what? Does this new transaction replace the old one in the mempool? Is it added separately? Also what does that mean for spenders of the original tx_out1 Alice constructed if the latter is the case? Wouldn’t that second transaction create another UTXO meaning the spender can create a tx_in that would have two choices of UTXOs to reference?
Edit: the specific output SIGHASH flag (in this case SIGHASH_ALL) isn't too important, the point was the creation of two txs with the same tx_in1. It seems to me that a UTXO might have two possible vouts and txids to choose from when holding the value of tx_in1, the first one being the UTXO that is created from Alice's tx and the second being the one created from Bob's. It seems to me only one of these txs can be added to the blockchain.

I think your confusion stems from you thinking of the original and the replacement transaction as the same transaction. However, they have different txids which uniquely identifies the outputs they create. These posts might help clarify: https://bitcoin.stackexchange.com/q/61613/5406, https://bitcoin.stackexchange.com/a/20855/5406

You are on the right track. Alice using SIGHASH_ALL | SIGHASH_ANYONECANPAY allows any other user to add additional inputs to the transaction. However, since Alice’s signature committed to all outputs (due to the SIGHASH_ALL part), the additional payers cannot change the outputs of the transaction, and their entire input will only contribute to the transaction fees.
Assuming that this new transaction increases the feerate and absolute fee sufficiently to displace the original transaction including any potential descendant transactions, it would invalidate the descendant transaction. Inputs commit to a specific UTXO explicitly per the outpoint (txid:vout). Since the addition of an input would create a new transaction that has a new TXID, the input to the original’s descendant transaction would cease to be expected to exist and as the original transaction is dropped from the mempool, the descendant would become invalid and be evicted alongside.
Since an input has to commit to exactly one specific UTXO it is spending, there are not "two choices of UTXOs to reference" for the descendant transaction. The recipient of the original/replacement transaction has to recreate the descendant transaction with the new input.
This rule would get an exception if the proposed SIGHASH_ANYPREVOUT were activated, which allows to identify UTXOs per their output script rather than their outpoint and would allow the recipient to create a descendant transaction that could spend the output of either the original or the replacement transaction.

Follow-up question: How do nodes know which transaction to drop? Why does the descendant transaction not have two different choices what to use as input?
When the original transaction is created, it might have a txid tx_orig¹. It’s first output (at position 0) then has the outpoint tx_orig:0. The descendant transaction input spends explicitly spends the UTXO with the outpoint tx_orig:0. When the replacement transaction with additional inputs is created, it has a different unique txid which might be tx_replace. Its first output would have the outpoint tx_replace:0. Therefore, the descendant transaction becomes invalid due to the original transaction becoming replaced.
Transactions are in conflict when at least one UTXO is spent by both transactions. Let’s say there was another previous transaction that created tx_ancient:1 and both the tx_orig and tx_replace try to spend that UTXO. The UTXO can only be spent once, so naturally including either transaction excludes the other from ever getting confirmed. Since only one transaction can be picked, miners pick the one that maximizes their reward. They hereby follow the replace-by-fee rules that require the replacement to exceed both the feerate of the original transaction and the absolute fees of all displaced transactions.

¹ In reality txids created by hashing the transaction data with SHA-256, so they are usually presented as 64-character hex strings.

First off, thankyou for the edit, this is my first post and I'll keep the formatting in mind. "Since an input has to commit to exactly one specific UTXO it is spending, there are not "two choices of UTXOs to reference" for the descendant transaction. " - what I meant is now we have two txs that have the same tx_in1. However it looks like this is solved by "dropping the original tx from the pool", but therein lies the question, how do nodes know which to drop? Its not like the original entry in the mempool will be queried by using the same txid, since as you mentioned, it changes.

If I'm a node and have received a tx message with id a72f…b1c9 it isnt immediately obvious to me how I should know to replace specifically the tx with id 1234…ab1e. I'll have a look at the BIP though.

Got it. Thanks alot!

I added the relevant parts of the comments to the answer and included a quick graphic to illustrate the exclusivity of the conflicting txs.

With SIGHASH_ALL, Alice’s signature commits to ALL outputs, which means if Bob changes or adds outputs to the tx, Alice’s signature will become invalid. This means Bob has no incentive to add an input to this transaction (unless Bob wants to send some sats to miners).
For the rest of the answer, I’ll assume you meant SIGHASH_NONE or SIGHASH_SINGLE, i.e. Bob is able to re-use Alice’s signature while changing or adding outputs.

Does this new transaction replace the old one in the mempool? Is it added separately?

By adding a new input, the hash of the serialized transaction will change (i.e. the transaction id), which means it will be a new transaction on the mempool. At this time, nodes will identify two transactions spending the same UTXO (Alice’s input) and will select one of them (in principle the one which pays more fees to the miners). The other one will be discarded.

Also what does that mean for spenders of the original tx_out1 Alice constructed if the latter is the case?

Depends on the SIGHASH used by Alice in her signature:

SIGHASH_NONE: the output can be changed or removed
SIGHASH_SINGLE: the output must remain as is in the transaction

Wouldn’t that second transaction create another UTXO meaning the spender can create a tx_in that would have two choices of UTXOs to reference?

No, an UTXO can only be spent once, so you cannot have 2 transactions spending the same UTXO.

SIGHASH_ALL | SIGHASH_ANYONECANPAY commits to all outputs but only to the one input the signature is on. See e.g. here: https://raghavsood.com/blog/2018/06/10/bitcoin-signature-types-sighash/


text:
Three functional tests in`test_runner.py` fail with `JSONRPCException: Method not found (-32601)`
I'm building bitcoind on MacOS 14.7.1 and running the functional test test_runner.py. bitcoind was built minimally with:
cmake -B build \
    -DBUILD_GUI=OFF \
    -DWITH_BDB=OFF  \
    -DBUILD_BENCH=OFF  \
    -DBUILD_FOR_FUZZING=OFF \
    -DBUILD_KERNEL_LIB=OFF \
    -DBUILD_SHARED_LIBS=OFF \
    -DBUILD_TESTS=OFF \
    -DBUILD_TX=OFF \
    -DBUILD_UTIL=OFF \
    -DBUILD_WALLET_TOOL=OFF

The codebase was prepared for the Chaincode Labs BOSS-2025 program, cloned from v28.0. The build completes without errors. However when running the functional tests from test_runner.py, three of the tests fail: mempool_persist.py, interface_bitcoin_cli.py and feature_rbf.py. All three tests fail with a similar error:
2024-12-03T11:09:18.858000Z TestFramework (ERROR): JSONRPC error
Traceback (most recent call last):
  File "/Users/maestrovt/workspace/github.com/chaincode/bitcoin-core-test-the-test-maestrovt/bitcoin/test/functional/test_framework/test_framework.py", line 132, in main
    self.run_test()
    ~~~~~~~~~~~~~^^
  File "/Users/maestrovt/workspace/github.com/chaincode/bitcoin-core-test-the-test-maestrovt/bitcoin/build/test/functional/feature_rbf.py", line 76, in run_test
    self.test_rpc()
    ~~~~~~~~~~~~~^^
  File "/Users/maestrovt/workspace/github.com/chaincode/bitcoin-core-test-the-test-maestrovt/bitcoin/build/test/functional/feature_rbf.py", line 622, in test_rpc
    self.init_wallet(node=0)
    ~~~~~~~~~~~~~~~~^^^^^^^^
  File "/Users/maestrovt/workspace/github.com/chaincode/bitcoin-core-test-the-test-maestrovt/bitcoin/test/functional/test_framework/test_framework.py", line 444, in init_wallet
    n.createwallet(wallet_name=wallet_name, descriptors=self.options.descriptors, load_on_startup=True)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/maestrovt/workspace/github.com/chaincode/bitcoin-core-test-the-test-maestrovt/bitcoin/test/functional/test_framework/test_node.py", line 936, in createwallet
    return self.__getattr__('createwallet')(wallet_name, disable_private_keys, blank, passphrase, avoid_reuse, descriptors, load_on_startup, external_signer)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/maestrovt/workspace/github.com/chaincode/bitcoin-core-test-the-test-maestrovt/bitcoin/test/functional/test_framework/coverage.py", line 50, in __call__
    return_val = self.auth_service_proxy_instance.__call__(*args, **kwargs)
  File "/Users/maestrovt/workspace/github.com/chaincode/bitcoin-core-test-the-test-maestrovt/bitcoin/test/functional/test_framework/authproxy.py", line 146, in __call__
    raise JSONRPCException(response['error'], status)
test_framework.authproxy.JSONRPCException: Method not found (-32601)
2024-12-03T11:09:18.918000Z TestFramework (INFO): Stopping nodes
2024-12-03T11:09:19.327000Z TestFramework (WARNING): Not cleaning up dir /var/folders/zh/y9jqq2qd6v5brxd0vdx6_rxh0000gq/T/test_runner_₿__20241203_060247/feature_rbf_125
2024-12-03T11:09:19.327000Z TestFramework (ERROR): Test failed. Test logging available at /var/folders/zh/y9jqq2qd6v5brxd0vdx6_rxh0000gq/T/test_runner_₿__20241203_060247/feature_rbf_125/test_framework.log
2024-12-03T11:09:19.328000Z TestFramework (ERROR):
2024-12-03T11:09:19.365000Z TestFramework (ERROR): Hint: Call /Users/maestrovt/workspace/github.com/chaincode/bitcoin-core-test-the-test-maestrovt/bitcoin/test/functional/combine_logs.py '/var/folders/zh/y9jqq2qd6v5brxd0vdx6_rxh0000gq/T/test_runner_₿__20241203_060247/feature_rbf_125' to consolidate all logs
2024-12-03T11:09:19.365000Z TestFramework (ERROR):
2024-12-03T11:09:19.386000Z TestFramework (ERROR): If this failure happened unexpectedly or intermittently, please file a bug and provide a link or upload of the combined log.
2024-12-03T11:09:19.386000Z TestFramework (ERROR): https://github.com/bitcoin/bitcoin/issues
2024-12-03T11:09:19.386000Z TestFramework (ERROR):

I'm a newbie to building and testing bitcoind, any suggestions appreciated!
After doing some searching I discovered that the problem was with my build configuration and specifically that wallet functionality was not enabled. When I rebuilt with the following cmake flags and ran the tests, all passed fine:
maestrovt@Edwards-MacBook-Pro bitcoin % cmake -B build \
    -DWITH_SQLITE=ON \          # Enable SQLite for descriptor wallets
    -DWITH_BDB=OFF \            # Disable Berkeley DB (legacy wallets not needed)
    -DBUILD_WALLET_TOOL=ON \    # Include wallet tool
    -DBUILD_SHARED_LIBS=OFF \
    -DBUILD_TESTS=OFF


text:
Find missing/masked characters from Bitcoin Private Key without knowing their location
I am giving the following (masked) BTC Private Key:
L2nBmWGfAH5Vf9re41VhG6BgPx25rxqGmkbaPtG14jAPtBUzhQab
What I can tell you is that the key above has 3 incorrect characters in it at different random spots. The incorrect 3 characters can be either the same letter or different, in both UPPER or lowercase.
But let's assume all 3 chars are the same letter in the same case (eg. A A A)
The alphabet has 26 letters with each having both uppercase and lowercase forms.
Considering that, we also know that the key is 52-char long.
What are the chances of someone cracking the key and successfully find the correct letters missing in it?
What calculations and permutations do we have to do ?
26x26x26...x26 , 52 times ?
If the exact location of the incorrect letters was known, I guess the calculations were much easier.
But in this case, the exact locations are secret because I try to understand how many permutations/chances are required to do, to find them.
How is it done? Is it even possible to crack the incorrect 3 missing letters or not?
Thanks
P.S -Forgot to mention, I own the full correct Private Key, I know what the missing chars are and where. I just wanted to clarify, I'm not trying to crack anybody's key. Plus the address is empty. I just want to know what are the chances of someone cracking the format and what the formula would be.

If you know which three characters are modified, you can find the correct input in 57³ = 185,193 attempts.
If you are looking at a 52 character string where each position can express as one of 58 different characters (base58), trying all combinations that modify this string in exactly three different positions would be calculated as follows.
Picking exactly three positions out of 52 would be modeled as an unordered sampling without replacement and we can calculate the 3-combinations of a 52-element set as a binomial coefficient of 52 over 3:
52! / (3!×(52-3)!) = 52!/(6×49!) = 52×51×50/6 = 52×17×25 = 22,100

In each of these 22,100 combinations of positions in the string, we would need to modify all three positions, and they can each take 57 alternative expressions than the original input. Therefore, there would only be fewer than 4.1 billion alternative input strings for us to test (4,092,765,300 to be exact).
Let’s apply that we know how the encoding works in addition, though. We are looking at a wallet import format (WIF) string which uses base58check. If I’m not mistaken, in its 52 characters, it encodes a 1-byte prefix, a 32-byte data portion, and a 4-byte checksum.
This yields the following additional restrictions:

The first character must either be K, L, or 5
The last eight bytes are a checksum calculated from the data. If I’m not mistaken, the checksum should contribute to the last six characters, and the last five characters entirely depend on the checksum.

This means that only 46 characters contribute to the data portion of the WIF string. If the first character is modified, it can only take two alternative expressions, and then there can be 0–2 changes in the data part. If the first character is correct, there can be 0–3 changes in the data part. We can model the optionality of a data character being modified by selecting 3 positions and allowing all 58 expressions. This will cause us to repeat the original string 15,180 times, but that’s not gonna slow us down much.
Picking two positions in the data part yields a mere 46!/(2!(46-2)!) = 1035 combinations. Each of the positions can take 58 expressions. With the two alternative values for the prefix character, we have to try only
2 × 1035 × 58² = 6,963,480

inputs to cover the case of a change in the prefix. Assuming a correct prefix and up to three changes in the data part, there are (46!/(3!(46-3)!) = 15,180 combinations of positions to check that can each take 58 different expressions, which leaves us with another
15,180 × 58³ = 2,961,800,160

possible inputs.
So, if you know that your string is modified in up to three places, you can find your original WIF in fewer than 3 billion attempts.
I’d guess that this translates to about two days of computation on a single core. If this is being considered as an additional security measure to obfuscate a backup, I would recommend using a different approach, since the footgun potential may outweigh the additional security.

Thank you for the explanation!
Basically, with 3 incorrect chars, we get almost 3 billion possible inputs, from which only 1 would be correct.
But if we have 4 incorrect chars this time, and apply the exact formula you did (to 46 char), we would get:
163,185 x 58⁴ = 1,846,682,399,760 (1.8Tn)

With 3 incorrect chars, we have 3 billion different private keys.
With 4 incorrect chars, we have almost 2 Trillion private keys.
The numbers grew exponentially, thus making it less likely for someone to find them.

Another fact that we must take in consideration is that you cannot even open a text file that big which contains all those possibilities.
Not to mention you wouldn't be able to import them into a wallet like Electrum for example, as it won't be able to process such a huge list.

Checksum in base58check is 4 bytes; WIF is either (now rare) 1prefix'80'+32 + 4 -> 51 char beginning '5H' through '5K' or (now usual) 1prefix'80' + 32 + 1flag'01' + 4 -> 52 char beginning 'Kw' through 'L5'. But I don't think this affects your result.

Ah yes, it’s a 4-byte checksum. Thanks for catching that. When you said that I was looking at it and realized: oh yeah, how would five or six letters out of 52 be enough to encode 8 bytes out of 41? :D

Well, with 4 incorrect chars we get almost 2 Trillion different private keys that we still have to test, from which only 1 is the correct private key. This is hard to check even for a cracker, so this should be pretty safu.


text:
Minerd http request failed: empty reply
I'm trying to run minderd with the following command (I'll replace my email address with EMAIL)
./minerd -o http://btc.kryptex.network:7777 -u EMAIL/worker -a sha256d -R 2

but getting the error "HTTP request failed: empty response from server" I tried to run it without http:// but same result.
I can ping btc.kryptex.network from the server. Also tried btc.kryptex.com.
What can it be?
Ubuntu 24.04.01 Xeon Gold 6148
Probably not very efficient for mining anyway I guess, but is there a better miner program to use to utilize the CPU above?

You cannot mine BTC using just a CPU, it is just orders of magnitude too slow. Even if there was a pool out there that supported such slow contributors, you'd make around ~2 sats per year with a 100 MHash/s CPU (a rough guess), or 2 USD per 1000 years. You need specialized ASICs.

@PieterWuille: Your comment would be fine as an answer.


text:
pubkey 33 or 65 byte
importmulti
   "success": false,
    "error": {
      "code": -5,
      "message": "Pubkey \"xxx\" must have a length of either 33 or 65 bytes"
    }

what its this

What do you want to ask exactly? Where is this code snippet from? People might not be in the same situation as you are and thus not understand what you are talking about.

I assume you are trying to use the importmulti remote procedure call (RPC). It looks like you provided something in the pubkeys key of the JSON array that did not adhere to the expected length of a public key.


text:
Transaction stuck between AddToWallet newupdate and AddToWallet update
This is my first question here, maybe I make a beginners mistakes.
I am running bitcoin core on a raspi, which is good with bitcoind but sometimes bitcoin-qt is too much load and the node breaks down. So I use bitcoin-qt only to spend money from my wallet, then shut it down and start bitcoind again.
Maybe I have shut down the software in a critical state. The transaction is not broadcastet. I have sent out another small amount which succeeded, and in the debug.log I can see, that for this one there are 4 entries:
AddToWallet txid newupdate
Submitting wtx txid to mempool for relay
AddToWallet txid
and later
AddToWallet txid update.
For the stuck transaction the last entry is missing in the debug.log.
abandon transaction tells me
Transaction not eligible for abandonment
listtransactions tells me
"confirmations": 0
since 2 days. The spent amount is substracted from my wallet content.
I am running Bitcoin Core version v22.0.0.
Have tried to update to version 27, but that would require OS update before.
What can I do?

Is it possible that your transaction is still waiting for confirmation because it’s feerate is not high enough to have bubbled up to the top?

The transaction does not make it to blockchain.com, so perhaps that's what I should have mentioned. I have learned to send a small amount with bitcoin-cli in the meantime, it does also not get the entry "AddToWallet txid update" but it is now already known in blockchain.com

If it doesn’t appear in the mempool of blockchain explorers, there might be another issue with the transaction. Are you doing something uncommon? E.g. is the feerate below 1 ṩ/vB, is too big, has too many op_return outputs, or is it otherwise non-standard?

The fee is "fee": -0.00001312. The two other transaction, which made it ti the mempool of blockchain explorers had similar fees. Another observation: Whenever I stop and restart bitcoind the entries in the log file appear again "AddToWallet" and "Submitting wtx txid to mempool for relay". Might it be the the software tries to complete the transaction, but the wallet is locked because of the passphrase?

After the transaction is signed, it is immutable, so that should not be a problem. 1312 sats should be enough if the transaction isn’t too big. Do you know the weight of the transaction? If you were able to share the txid or the makeup of the transaction, we could understand better what the problem might be.

Here is the output of gettransaction, I have repaced some values by xx, because it is dealing with money. Please let me know if any of these values are necessary for the analysis.

I deleted your comment because it still contained the txid. Here is what Horst wrote with a bit more sanitation:
"amount": -0.xx, "fee": -0.00001416, "confirmations": 0, "trusted": true, "txid": "xxxxx", "walletconflicts": [ ], "time": 1733502999, "timereceived": 1733502999, "bip125-replaceable": "yes", "details": [ { "address": "xxx", "category": "send", "amount": -0.xx, "label": "xxx", "vout": 1, "fee": -0.00001416, "abandoned": false } ], "hex": "xxx"

I see that the transaction only has a single output, but I don’t see the input count. One thing that is missing is the weight of the transaction. Without knowing the size/weight, the fee doesn’t tell us much about the feerate. If the feerate is under 1.99 s/vB, most mempools will currently probably not accept the transaction and not relay it, so that could explain why it is not making it to blockchain.com.

Thanks a lot for the explanations so far. The size of the contents of the "hex": value is 445 characters. Would it be a good try to do sendrawtransaction with this hex-value?

That should be more than 3 s/vB, and enough for the transaction to propagate across the network. Yes, you could force your node to rebroadcast it by using "sendrawtransaction" with the hex of your transaction. Is it possible that it has a parent transaction that did not propagate, or even a chain of ancestor transactions that it depends on? Or is it maybe an issue with blockchain.com?—Is it visible on mempool.space? The most likely explanation seems to me that the one of the inputs is unknown to other nodes, or the tx is in some way non-standard. But if it’s just a payment that would be odd.

Oh, is the recipient amount very small?

This conversation is getting pretty long. I’ve moved these comments to chat; please join the chat and continue the discussion there, if you are able.


text:
What is an actual bitcoin worth "blockchain"
Blockchain bitcoin was founded in 2009 is that correct?
What about the blockchain?
Where?

Hey Luke, I’m afraid that I don’t understand what you are asking. Could you please edit your question to make sure that it is easy to understand what you want to know more about?


text:
Why do most HD wallets require choosing either all Segwit or all Legacy but Bitcoin Core can support both simultaneously?
Electrum and the mobile wallets I've used can only support one type of address per HD wallet, but Bitcoin Core can support both at the same time.
Is that simply a choice that Bitcoin Core made or is there something unique about Bitcoin Core in this case since it's a full-node wallet?

Do you mean having one key be multiple address types, or the wallet only gives out addresses of a single type? Regardless, Core being a full node is unrelated.

By "key" do you mean "seed"? Yes, I think I mean that Bitcoin Core is able to give different types of addresses per seed while other wallets can't.

It's not true, there are some mobile wallets like Samourai that support P2PKH, P2SH and segwit addresses derived from the seed.


text:
How to build a portable bitcoind binary for macOS?
Bitcoin Core ships a portable bitcoind with the Linux and Windows releases, but doesn't ship one with macOS. And I couldn't find instructions on how to build one.
Can someone provide step-by-step instructions on how to build a portable bitcoind binary for macOS?
The host platform can be Ubuntu or macOS. I have access to both.
(Just to clarify, the goal is to have a self-contained bitcoind binary that runs on different macOS machines without having to install additional dependencies)

The bitcoind provided by the install should be portable. Have you tried installing Bitcoin Core on one machine and then just copying the bitcoind binary?

The Bitcoin Core macOS install doesn't have a bitcoind in it

Oh I didn't think to check the tar.gz file. It does contain the files I need. I suspect those will be portable but I haven't tried yet! Thank you!

macOS portable binaries are uploaded to bitcoincore.org. On the download page, under Mac OS X is a link to tar.gz. That tarball contains the macOS portable binaries.

Bitcoin Core's release process using gitian does produce these portable binaries. So you can replicate the same process to build your own portable binaries. You can read about how to do that here.
There is a script, contrib/gitian-build.py that should do everything for you. It should work on Ubuntu. It will install all of the necessary dependencies and setup the gitian environment before doing the build when you run it with the --setup option.
Your build command with this script would look something like
bitcoin/contrib/gitian-build.py -Dnb -o x <name> 0.20.0

Use the -h option to get usage information
-D tells the script to not try to sign the resulting .assert file that contains all of the build hashes.
-n tells the script to not try to commit the .assert file that is created.
-b means to do a build
-o x tells the script to specifically do a macOS build (-o means operating system, x for OSX)
<name> that's where you would put your username. This is just for identifying signers who participate in the gitian build process. You can just set this to be whatever you want.
0.20.0 is the version to build, in this case, 0.20.0. You can put whatever other version you want.
The script will create a directory named bitcoin-binaries. Inside of that will be a directory with the version name. And inside that directory will be 2 files - a .tar.gz file and a .dmg file. The .tar.gz file contains the portable binaries that you can use.

Additionally some gitian builders (like myself) upload all of their gitian build results so you could just download those files.
Personally, I upload my results to github. The macOS portable binaries are named bitcoin-<version>-osx64.tar.gz.
Jonas Schnelli also uploads the results of his builds. You can find them at on his website.
In both cases, the files that you download should have hashes that match those of other gitian builds. The gitian builders commit their hashes to this github repository so you can go there to check that the file you download has the same hash that the gitian builders computed.


text:
Switching between pruned and non-pruned blocksdir ok?
Is it ok to swap the nodes blocksdir upon restart without corrupting sth. cache etc.. I have two blocksdir: one locally pruned blocks, the other external with all blocks in blocks on usb drive. Id like to swap to external HDD blocksdir to eg rescan etc.
 like bitcoin-qt -data-dir=stays_same -blocksdir=changes


text:
Error code -3 when signing message from gui?
I tried to sign a message from gui v18 and the debug console without success:
With the popup it shows red text:

"the entered address does not refer to a key. Please check the address
  and try again"

With the console:

"Address does not refer to key (code -3)"

the approach was:
- get a fresh public address from the wallet (new receive address)
- paste it into the address field
- type 'foobar' into the message field
- click sign message
in the console i unlocked the wallet (for long enough!) then
`signmessage "pupaddr" "foobar"
both with no success. I double checked the pubaddr (taken from the address field on the right button).

the throw happens here in the sourcecode. The addr. starts with 3... .

The message signing feature in Bitcoin Core only works for P2PKH addresses (which start with 1...). There isn't an established standard for other ones (a few competing proposals exist, though).

For how long will i abe able to use this feature for  P2PKH to either generate or verify those adresses? currently i do getnewaddress "mylabel" "legacy"


text:
Background information on warning, when closing wallet in pruned mode
What are the implications and reasons behind this warning on gui popup, when closing the wallet:

"bitcoin closing the wallet for too long can result in having to
  resync the entire chain if pruning is enabled"

A pruned node will always catch up, whats the hassle?

A pruned node will always catch up, whats the hassle?

The issue stems from having Bitcoin Core opened but the wallet in question is not loaded in Bitcoin Core (because you closed it). Bitcoin Core will continue to be synced and receive new blocks and transactions. However, it will discard old blocks once it has stored the configured amount of block data.
Suppose, for example, your pruned node is synced to block 570000 and is configured to keep 1000 blocks of data. When you do Close Wallet at block 570000, your wallet is synced to block 570000. But Bitcoin Core is still running and receiving new blocks and transactions. 1000 blocks later, at block 571000, block 570000 is discarded. One block later, block 570001 is discarded.
Now if you try to load your wallet again, it is going to try to catch up to the latest block, so it looks for block 570001 to begin rescanning. However your node has already discarded the block, and thus the rescan is unable to continue. In order to get block 570001 to scan it, you will need to redownload the entire blockchain. This is what the warning is for.
During the entire time your wallet was closed, Bitcoin Core was still running. However your wallet was not loaded into it, so it does not know to scan for transactions that belong to that wallet, and even if it did, it wouldn't know where to put the transactions it finds. So it is always catching up, but without the wallet in question loaded, it isn't finding the transactions that you care about.

so as long as i keep the same wallet open nothing is missed and all ok?

Yes. With pruning enabled, you shouldn't really be dynamically loading and unloading wallets.

If you have a wallet that isn’t opened for the period of pruning, it won’t be able to scan transactions. If you set the limit to 2GB, and 4GB of blocks pass without opening the wallet, it may miss incoming transactions as the required blocks will have been deleted.


text:
Parse signature and public key from scriptsig in an input transaction of bitcoin protocol
I'm interested in bitcoin protocol and checking validation of a transaction.
in a TX message, we can see one or more than one input transaction and every input transaction has a previous transaction and a script.
As you know probably, the script contains two components, a signature and a public key. The public key must match the hash given in the script of the redeemed output. The public key is used to verify the redeemers signature, which is the second component.
I want to parse signature and public key at first step, and in the following I want to check the validation if transaction from public key.
anyone can help me?


text:
Convert 33-byte public key with no parity prefix to address
I'm writing a blockchain parser and I keep running into this type of scriptPubKey in the Litecoin blockchain:
20a75ca72ffd994d2004d67b0e89015913f7352455d0111ede590430037c9fe2ac
20e664c3f6909687499d9bf13108e35306477a8d71b20655a75cbc64270416a9f2
20651b85139631645bfd68327d85af913a7e33e74433848c5facfe48368c7d1504
20604f5ced8c595af687a0e8718098c4818b7d51f2ba79cca931dc74d6cdb8c021

The length suggests they might be compressed public keys but there is no parity byte. I can look at a few dozens of these and all bytes vary, except for the prefix 20 which is the length.
Blockchair seems to parse these as 's-something' addresses, like this:
https://blockchair.com/litecoin/address/s-31fa6bd469e97be4fe639911ca60bddf
https://blockchair.com/litecoin/address/s-4e7d283f055044b547f571bfdbbdb291
What are these addresses, and how is it doing that? I assume they artificially add the s- because dash in not in base58, but I can't get anything to match the address to the right of it either.
I've tried the regular way of compressing other formats like this:
temp = prefix + hash160(scriptPubKey)
address = base58Check(temp + checksum(temp))

With and without checksum, with and without hash160.
Even scanning for all possible prefix bytes I can't get the address to the right of the dash.
I've discovered that I have to base58 something with 23 bytes to get the target length, but that's about it.
Other parsers can't seem to decode this address (see links on blockchair page itself), only blockchair.

These are not public keys. Not everything that is 33 bytes is a public key. In this case, it is a 32 byte hash. The first byte indicates that the next 32 bytes are to be pushed to the stack. So the result is that the actual data in this scriptPubKey is a 32 byte string.
In this case, this particular transaction is a coinbase transaction in a block mined by p2pool. This particular script is some p2pool specific data and is a hash, not a public key.

These are not public keys, they are data or bitcoin script fragments.
For example, the address s-38f26094a4e3514933fc2bf56a1f2f26 is actually the bitcoin script:
OP_IFDUP OP_IF OP_2SWAP OP_VERIFY OP_2OVER OP_DEPTH

Addresses are a human construct, and only certain well defined bitcoin scripts (p2pk, p2pkh, p2sh, p2wpkh, p2wsh) are convertable to addresses. All other scriptPubKeys (or locking scripts, if you prefer to call them that) are non-standard outputs, and are interpreted by different explorers in different ways. Blockchair seems to prefer a s- prefix followed by some kind of hash of the raw script.

That is a known case, and I'm getting those too. That is a bug, and those op codes translate to the ascii word "script" according to this article: https://medium.com/@alcio/a-look-at-bitcoin-non-standard-outputs-c97f65cccbb6. This is a different situation though, as the pubkey is clearly hashed, and there are hundreds and thousands of those constantly. It must be some kind of standard format, and even its a script, it must be decodable so that it can be verified in some way. I'd be ok with having just the plaintext script


text:
What is Profit Ratio in BTC.com mining calculator
BTC.com website has a sophisticated Bitcoin mining profitability calculator and a simple calculator. The sophisticated one has a lot of settings.
There is a particular setting named "profit ratio" which is a percent.
I experimented with it and found that when this is equal to 100% the income is computed exactly correct and when I increase it the income is increased by that amount of percent. For example in 110% the income is calculated 10% more.
The default value is 102%.
What is the use of this setting?
https://btc.com/tools/mining-calculator


text:
How does Lightning multipart payments (MPP) protocol define the amounts per part?
From the source code of LND, payments are split in half and retried if no path for the full amount is found:
func (p *paymentSession) RequestRoute(maxAmt, feeLimit lnwire.MilliSatoshi,
    activeShards, height uint32) (*route.Route, error) {

        ...
        // This is where the magic happens. If we can't find a
        // route, try it for half the amount.
        maxAmt /= 2
        ...

For normal payments, I understand the algorithm will split the payment until it fits a path, send it, and update the weights in the graph to avoid reusing the same path. Then we can do the same for the remaining amount. This is confirmed in this answer.
However, for MPP, it seems there is a set of parts per round, which are sent concurrently. If this is the case, how does MPP define the amount of each part before sending the parts?

This is not specified by the protocol. Implementations can do what ever pleases them. For example I believe many would use a more or less uniform split into parts given the amount and the number of parts they wish to send.
This form of splitting is discussed in this research paper and was shown to be not optimal. The previous link also provides the optimal solution by recognizing that payments should be understood as minimum cost flow problems. However even dissecting the resulting payment flow into shards is arbitrary and not well defined. The typical solution is to use a chain of cascading shortest paths as done in this function.
Newer research is in progress that uses augmenting paths which are saturated such that the probability of the amount for a part reaches a certain threshold.
Disclaimer: I am the main contributor of all the links provided


text:
Manipulation of Bitcoin RPC with programmatically
I'd like to use Bitcoin source code(only RPC part) to manipulate with programmatically.
I have bitcoind and bitcoin-cli installed in my local computer but I don't know how to write code for my purpose.
What I want to get senario is Writing the code that works like bitcoin-cli getblock <hash>
Actually, I got same result by calling shell script code like below.
#include <iostream>
#include <stdlib.h>

int main() {
    system("bitcoin-cli getblock 000000000000000082ccf8f1557c5d40b21edabb18d2d691cfbf87118bac7254"); // Get 300000th block
    while(1);
}

However, I want to write c++ code using pure Bitcoin source code.
To achieve the my goal,

I should know how to call getblock() function.
I should know how to compile it.

getblock function is defined here : https://github.com/bitcoin/bitcoin/blob/ae6943620ab369e8e0865c3ec2d2848ba9389325/src/rpc/blockchain.cpp#L828
If you have experience on my question, can you give some guide?
Sorry for my English.

Dupclicate of https://bitcoin.stackexchange.com/questions/43937/how-to-use-json-rpc-in-c-in-linux/88031


text:
How to generate new address wallet ? I want to use it per customer
I want to create an exchange site using API.
How can I check that my customer had a deposit without using a transaction number?
I want to create a new address wallet per customer how can I do it?
I'm looking for a wallet that can generate new addresses

First of all, there are two ways to accomplish that.

Create an account at http://www.block.io, generate an API there and they will give you some endpoints to query the blockchain, generate new addresses for your wallet, broadcast transactions and so on.
You can create your own endpoint if you run a full node of bitcoin, there you can create your own transaction, query the mempool for unconfirmed transactions, query blocks to find out if your address is in any of them.

The second way is kinda tricky since you will need to understand how the transactions are stored in the blocks, maybe some parsing, but if you are starting I really recommend you to stick to the first way.
Developing an exchange is complicated, there are a lots of factors to account for. When I was developing mine, I got several issues with transaction checks and confirmation.


text:
Cannot open dump wallet file (code-8)
Is anybody familiar with this problem using btc qt debug console?
Would like to dump wallet and sweep into electrum.

What is the exact command you're using?

Dumpwallet. To a created txt file on my C drive. Don't know what the problem is. What does code -8 mean? Should I use a certain type of txt file? Should I stop syncing my node while doing it? Any help would be great.

Well you need to pass the file name to dump to, how are you soecifying that?

In the debug console i type Dumpwallet C:......

Should I stop syncing? I'm getting a new error now. No longer "code-8"

You need to enclose the filename in quotes.
eg:
dumpwallet "C:\output.txt"

As opposed to:
dumpwallet C:\output.txt      (wrong)


text:
How to connect the Blockchain to the Network without IP Address?
I would like to know how do we connect the blockchain to the network when there is no IP?
Indeed we all use bitcoin core (for bitcoin) to connect and retrieve information and update.
However the source code of bitcoin core (I have never read it) does not redirect us to an IP address, but people still manage to find themselves, without ever saving the IP.
How can people know or find the information, when the source code of the software does not have IP, they do not know address (IP) which has the information?
Knowing that the IP is the basis of the connection ...

The bitcoin client has many means to discover other ip in the bitcoin network, in the case of DNS request, which will return many ip address each times, here are the diferent DNS used as in 2017:

seed.bitcoin.sipa.be dnsseed.bluematt.me dnsseed.bitcoin.dashjr.org
  seed.bitcoinstats.com seed.bitcoin.jonasschnelli.ch
  seed.btc.petertodd.org

You can see all the methods used by the bitcoin client to discover nodes here: https://en.bitcoin.it/wiki/Satoshi_Client_Node_Discovery

if this is the case, it means that bitcoin core will no longer work for new clients (who have never started it), if the DNS is unavailable.
Is it a certain way of centralization? : /

If all those hardcoded DNS became unavailable, you will soon see all kind of post on every media, like facebook or even here posting new DNS to connect to, there is even nodes in space now https://cointelegraph.com/news/bitcoin-in-space-blockchain-satellite-receives-first-transaction good luck shutting all of them down ;)

OK i understand, very thank !


text:
Understanding assurance contract in bitcoin scripts
I'm having a bit of trouble understanding example 3, the bit coin assurance contract on the bitcoin wiki(https://en.bitcoin.it/wiki/Contract) where senders choose to donate money only if 1000BTC total is raised. Here is the description of the assurance contract example:

An entrepreneur creates a new address and announces that the good will be created if at least 1000 BTC is raised. Anyone can contribute.
Each party wishing to pledge creates a new transaction spending some of their coins to the announced address, but they do not broadcast it. The transaction is similar to a regular transaction except for three differences: Firstly, there cannot be any change. If you don’t have any outputs of the right size, you must create one first by spending to one of your own addresses. Secondly, the input script signature is signed with SIGHASH_ALL | SIGHASH_ANYONECANPAY. Finally, the output value is set to 1000 BTC. Note that this is not a valid transaction because the output value is larger than the input value.
The transaction is uploaded to the entrepreneur's server, which saves it to disk and updates its count of how many coins have been pledged.
Once the server has enough coins, it merges the separate transactions together into a new transaction. The new transaction has a single output that simply spends to the announced address - it is the same as the outputs on each contributed transaction. The inputs to the transaction are collected from the contributed pledges.
The finished transaction is broadcast, sending the pledged coins to the announced address.

where each transaction is of the following form from bitcoin wiki(https://en.bitcoin.it/wiki/Transaction):
Input:
Previous tx: f5d8ee39a430901c91a5917b9f2dc19d6d1a0e9cea205b009ca73dd04470b9a6
Index: 0
scriptSig: 304502206e21798a42fae0e854281abd38bacd1aeed3ee3738d9e1446618c4571d10
90db022100e2ac980643b0b82c0e88ffdfec6b64e3e6ba35e7ba5fdd7d5d6cc8d25c6b241501

Output:
Value: 1000
scriptPubKey: OP_DUP OP_HASH160 404371705fa9bd789a2fcd52d2c580b65d35549d
OP_EQUALVERIFY OP_CHECKSIG

I'm having trouble understanding step 4. If the entraprenuer only ends up raising 500BTC and never meets their goal of raising 1000BTC why can't the entraprenuer change the output value to 500BTC, then broadcast the transaction and cash out anyway without each of the the sender's consent?

SIGHASH_ALL | SIGHASH_ANYONECANPAY signs the outputs. So the entrepreneur cannot change the output, as it would invalidate the signatures on all the inputs he received.
There is another way he can bypass this though - if he has 500 BTC himself (or can borrow that amount very briefly) - he can contribute that himself, getting away with raising only 500 BTC but still taking the coins. The only way around that is having some form of escrow that verifies the payments.


text:
How exactly does the network know if the miner has put forth a valid proof of work?
If all that the miner needs to do is find a hash with a certain number of leading 0's of the block's header, couldn't they just find a random hash with the required number of 0's that isn't necessarily the hash of the block header? How would the nodes know?
How exactly do the nodes decide if the proof of work is valid? What protocol is used for this?

I had to do more research to understand how the network prevents this kind of thing because these answers aren't clear at all. Basically what I understood is that the node will recalculate the hash with the given nonce and see if it matches the one put forward by the miner.

the key is that there is no ‘hash put forward by the miner’. The miner puts forth the block, and all other nodes use it (the block’s header) to calculate the hash themselves, and ensure it is valid. The block does not explicitly contain its own hash.

What you are saying is that the nodes have to do the work to find the valid hash all over again. It doesn't make sense and I know it's not how this works. The question that you say is a duplicate also doesn't provide clear answers on how exactly does the network prevent this.

no, the nodes do not have to do the work again, you misunderstood my comment. The network’s nodes simply hash the header of the newly proposed block, and thus determine for themselves the validity of that block. If it is valid, they will quickly and easily determine this to be true. If it is invalid, they will discard it. No iterative proof of work required, and no opportunity to just ‘propose a random hash’, as you suggested.

@Sara A miner tries to mine billions of possible blocks until they find one that works. They then propose that block and everyone else just has to try that one block one time to see if it works or not.

When a miner authors a new block, the block contains a block header. The block header is used to put the new block in relation to previous blocks and gives a summary of the transactions. In detail, the block header contains:

a version byte
the block hash of the previous block
the Merkle root that commits to all transactions in the block
a timestamp
a field corresponding to the current difficulty
a nonce that provides some randomness.

This block header has only 80 bytes of data. When these 80 bytes are put through SHA-256d once, they result in a hash which we use as the unique identifier of the block, the "block hash". The block hash is used to announce the block to peers and in the next block to commit to this one. This same "block hash" interpreted as a number has to meet the difficulty requirement ("have a certain number of leading zeros").

What you are saying is that the nodes have to do the work to find the valid hash all over again.

Not quite! Finding a block header for which the hash meets the difficulty requires a multitude of tries. Since it is impossible to predict which block header will succeed except by hashing and checking, the fact that a valid block has been found proves that a large number of tries have been expended (about 4,92×10²² in average currently). We call this concept proof-of-work. However, the validation of the work only requires to put the provided block header once through a hash function and then to compare the resulting hash with the difficulty which is trivial to do!
In fact, this is the first of a number of validity checks that other network participants do when they receive a new block candidate.


text:
Cash app Bitcoin transaction stuck on the blockchain for over 2 months. What should I do?
Yes I had a withdrawal from wild.io casino sent to my cash app where it was rejected and reversed with a 42-cent fee it has now been 2 months and 4 days since it was reversed!! I have went as far as paying a couple people and a site to get my Bitcoin transaction unstuck and was very unsuccessful and just got my money taken!! I am fairly new to all this cryptocurrency stuff and cash app won't give me nothing no information on what to do and wild.io can't do nothing about it either!! Can somebody please give me some insight on what to do if I have any options!! Somebody please help me out with this I really could use the money right now in the worst possible way!!


text:
Easiest way to communicate with Bitcoin core with terminal (not bitcoind and without bitcoin core's terminal)?
I have bitcoin core running. I installed it through bitcoin.org and installed via a dwg file. I want to use bitcoin-cli commands through my mac's terminal app. (not with bitcoin core's terminal). I tried cd into bitcoin core's file(which is in ssd because the lack of available space) and tried bitcoin-cli but it didn't work. Is there a way to talk to my bitcoin node through terminal (not with bitcoind)
Basically, I want to do bitcoin-cli operations from my mac terminal after running bitcoin-core

This appears to be a dependency problem. It may be solved by upgrading to 22.0.

I upgraded but it didnt solve the problem. Isn't there a way to speak to bitcoin-qt from operating system's terminal but not bitcoin-qt's console?

Sure, it's just JSON-RPC, there are frameworks in most programming languages that speak that. But this is a bizarre result. Bitcoin Core doesn't use openssl anymore, so I don't see why it is depending on it. Is there any chance you have an old bitcoin-cli, and it's using that?

Uninstalling and reinstalling openssl library (via homebrew) solved the problem somehow.


text:
What is the best way to manage a lightning node to develop a paywall app with c-lightning?
I want to develop a lightning powered paywall app with Django on heroku but don't know how to implement lightning part of the application besides using a rpc lightning package which needs a lightning node address which is online constantly. I am currently running a bitcoin node and a c-lightning node on my laptop but I shutdown my computer at nights(electricity costs too much in Turkey). So what should I do? Should I switch to a seperate lightning node like Umbrel which runs 24/7 without the need of a laptop, install lightning node on Heroku(I dont know if it is possible) or use 3rd party providers like btcpay-server(if it is possible)?

Accepting LN payments requires being online because the protocol is interactive.
Running pre-configured hardware nodes like Start9 Embassy, Nodl, Raspiblitz or Umbrel is relatively easy way to get there.
Another way is to host your lightning node on a VPS server online, similar you would host your website.

Yea if you are looking to take lightning payments having your own node is ideal. probably you wouldnt run it on heroku but you could run the code that interacts with your node on heroku easily. Unfortunately I can't point you to a working application of this but generally all you need the program to do is generate a lightning invoice in the amount the user needs to pay and then send that to the users endpoint. It should be relatively simple. Only security feature might be anti-spam so people dont flood the node with requests for invoices. Checking for payments is a bit more tricky i suppose, for that you will need to assign users to their invoices in a way that you can depend upon.


text:
secp256k1 recovery id and how does the code work
Here I have some questions regarding the recovery id (recid) defined in secp256k1 implementation.
I had found few posts about how the recid is defined and what does it mean, also the source code (https://github.com/bitcoin-core/secp256k1/blob/master/src/modules/recovery/main_impl.h).
As explain by bitcoin developers like Peter Wuille the recid is [27~30] for compressed pubkey and [31~34] for non-compressed pubkey.
my question are:
1). from the secp256k1 code I did not find 27~34 value, all I found are 0~3; so where the difference come from?
2). I guess [27~30 and [31~34] such value set does not come from nonsense, but why using these two value set to map to [0~3]? Guess some tricky hidden inside--is that because of some trick on bit-level operation?
3). can someone help to explain this piece of code (come from https://github.com/bitcoin-core/secp256k1/blob/master/src/modules/recovery/main_impl.h#L104-L112)
looks for me recid &2 (0x10) actually select recid=2/3 case.
and if recid !=2/3 all left code will be executed--is that understanding correct?
if (recid & 2) {
    if (secp256k1_fe_cmp_var(&fx, &secp256k1_ecdsa_const_p_minus_order) >= 0) {
        return 0;
    }
    secp256k1_fe_add(&fx, &secp256k1_ecdsa_const_order_as_fe);
}
if (!secp256k1_ge_set_xo_var(&x, &fx, recid & 1)) {
    return 0;
}

If you have one specific question, there should be one question mark, not four.

Sorry for that--I think initially I had one "specific" question but when I was trying to draft the question, some more questions may be raised and it turned out better to make them seperately..

1). from the secp256k1 code I did not find 27~34 value, all I found are 0~3; so where the difference come from?

The serialization as a 65-byte signature + recovery byte happens on the Bitcoin Core side, not in libsecp256k1. See key.cpp:
vchSig[0] = 27 + rec + (fCompressed ? 4 : 0);

2). I guess [27~30 and [31~34] such value set does not come from nonsense, but why using these two value set to map to [0~3]? Guess some tricky hidden inside--is that because of some trick on bit-level operation?

There is nothing special about it. I just picked an arbitrary offset so the signature byte arrays wouldn't be confused with public keys or other signatures. But there is no special meaning to the number 27 (and in retrospect, it's a rather strange choice).
I'll try to expand this answer later to explain the code.

Great thanks for the reply, didn't expect to get the original author helping on this;-) ... it took me some time to think why this number is selected. Waiting for more comments of the code.. for my Q3, from some other post I thought it would be due to when recid=2/3 means (x,y) sit on the location that satisfies (n<x<p)? But I did not understand the function of secp256k1_fe_cmp_var yet.

if possible, could you please help to explain the logic behind the if secp256k1_ge_set_xo_var(&x, &fx, recid & 1)? this piece of code looks not link to SEC standard document-- is that specific tricky to the secp256k1 curve?

Recoverable pubkeys are something we invented for bitcoin message signing. It's not in any standard.

I can find the pub key recovery method here: https://www.secg.org/sec1-v2.pdf (CH4.1.6) but the detail implementation is not the same (it has no recover id) a recover id indeed help speed up the algorithm, but short of the reference of the algorithm--is there any more documentation can help to understand the logic behind these piece of code?

I paste one more post below, put all my understanding here if you can help to comment whether they are correct or not... if anything is wrong--would you please suggest the correct understanding?

Put some comments embedded with code, still, some questions remain, hopefully, to be improved time by time.
The below code used in public key recovery method, is aiming for recover the R (rx, ry) from provided r and recid (0 or 1), for recid (2or3) case, simply reduce (provided r)-n.
{    if (recid & 2) {    /* if recid=2 or 3 */
    if (secp256k1_fe_cmp_var(&fx, &secp256k1_ecdsa_const_p_minus_order) >= 0) {
        /* if fx >= p-n, not a valid signature */
        return 0;
    }
    /* else if fx < p-n, do fx=fx+n  */

    secp256k1_fe_add(&fx, &secp256k1_ecdsa_const_order_as_fe);
}

next call up secp256k1_ge_set_xo_var(), for the recid=0/1 (recid 2/3 result will also be passed in?)
static int secp256k1_ge_set_xo_var(secp256k1_ge_t *r, const secp256k1_fe_t *x, int odd) {
/* for all below operation on Field (p), implicitly mod p */
secp256k1_fe_t x2, x3, c;
/* directly get rx */
r->x = *x;
/* calculate x2 = x^2 */
secp256k1_fe_sqr(&x2, x);
/* calculate x3 = x^2 *x = x^3 */
secp256k1_fe_mul(&x3, x, &x2);
/* set up infinity flag =0 as won't be 1 for below operation */
r->infinity = 0;
/* calculate c=x^3 +7 */
secp256k1_fe_set_int(&c, 7);
secp256k1_fe_add(&c, &x3);
/* calculate sqrt(c), put to ry*/
if (!secp256k1_fe_sqrt_var(&r->y, &c)) {
    return 0;
}
/* what does normalize mean ? */
secp256k1_fe_normalize_var(&r->y);
/* odd (recid passed in) could be 0(ry=even) or 1(ry=odd) */
/* exp: if odd=0, but odd(ry)=1 !=0, then do reverse*/
/* exp: if odd=1, also odd(ry)=1 == 1, then simply return */
if (secp256k1_fe_is_odd(&r->y) != odd) {
    /* for getting -R, i.e. -(r.y) one can simply do p-(r.y) */
    /* below likely does p*2*(1+1)-(r.y), why??*/
    secp256k1_fe_negate(&r->y, &r->y, 1);
}
return 1;

}

Everything is implicitly mod p. We set infinity=0 because we need to set infinity=0 or infinity1 in the output, and we know the result won't be infinity. Normalize means bring to canonical representation (secp256k1_fe can be temporarily "denormalized" which allows some faster operations, but isn't compatible with fe_is_odd, for example). fe_negate(a,b,n) just computes the negation of b and puts it in a; the n argument gives the maximum magnitude of the input (which is 1, because the input is normalized).

For dealing with recid=2 or recid=3, that means we have r = R.x - n (remember: r = R.x mod n, so either r = R.x or r = R.x - n). We're given r in the signature, but we need to recover the full R.x coordinate. If recid=2 or recid=3, we need to add n: R.x = r + n.

The deal with normalization: field elements are represented as 5 52-bit "limbs" (on 64-bit symbols). Those limbs are allowed to overflow (be larger than 2^52), which permits addition/negation without carry (much faster for some operation); for addition you just add the corresponding limbs; for negation you subtract it from a multiple of limbs representing n... but you need to know how big the input is allowed to be (as you need to make sure no limb can result in a negative number). That's why fe_negate takes in a magnitude argument.

@PieterWuille "Everything is implicitly mod p“ -> this is for the F(p) operation I suppose? if operate in Group (for exp, calculate Point Add/Mult, which are operated on both X/Y coordination, that will be mod n?

The scalar factor in point multiplication is indeed modulo n. But all variables in this snippet ("fe" types) are modulo p.

@PieterWuille for recid0/1 case, I have two questions: Q1: we need to calculate the sqrt(x3+7) to get the Ry -- if solving a mathmetic function we shall get +/- solution-- from code it likely we will get only one solution--that could be old or even, we then compare old/even with recid to decide if we need do negate.--is this correct?  Q2: if I don't use the secp256k1 speedup lib (which has 5*64bit and mentioned normalized tricky), but using some general 256 bit operation, if I know R.y, to getting negation -R.y, can I do the (p-R.y)?

Yes, computing the sqrt(x^3+7) always gives you the square root that is a square itself, but that could be even or odd. By comparing its parity with recid and conditionally negating, you get the intended result. 2) Yes.

@PieterWuille I noticed that the signature (r,s,z,x,y) generated by OpenSSL API will be verified fail by using secp256k1 lib, further investigate I found that I need to reverse the byte sequence for each of (r,s,z,x,y)--then it will pass. It seems the secp256k1 lib arrange the storage differently (byte-level reorder) than the OpenSSL, is that correct? and if that is true, why?

That's not correct, and I suspect you're not using things right. The serialization of public keys, signatures, and private keys all uses big endian, both in libsecp256k1 and OpenSSL.

@PieterWuille I had made one successful public key recovery routine by following the secp256k1 lib implementation method. Thanks so much for your question answer. I find most of the timing consumed on the recovery y from the y^2 (I simply calculate the power ((p + 1)/4) %p by using the fast exponential modular algorithm, which usually takes 256 iterations) -- if refer to lib secp256k1_fe_mul that looks taking some fast computing (multiple squares and multiply, though underline are assembly code) --is there any reference documentation that can you recommend for reading?


text:
secp256k1 lib compiling issue "invalid use of incomplete type secp256k1_context"
I make a small test code (test.cpp), try to show the problem I meet:
to compile use
g++ -o test test.cpp -lsecp256k1
#include <stdio.h>
#include "secp256k1.h"

static secp256k1_context_t *ctx = NULL;

int main(int argc, char **argv) {

ctx = secp256k1_context_create(SECP256K1_CONTEXT_VERIFY | SECP256K1_CONTEXT_SIGN);
printf("Context created!\n");
secp256k1_context_t *ctx_clone;
ctx_clone = secp256k1_context_clone(ctx);
printf("Context cloned!\n");

/* if enable below two lines of code the compliling error will happen */
// secp256k1_ecmult_context_t* clone_ecm_ctx;
// secp256k1_ecmult_context_clone(clone_ecm_ctx, &ctx->ecmult_ctx);

secp256k1_context_destroy(ctx_clone);
secp256k1_context_destroy(ctx);
printf("Context destroyed!\n");

return 0;
}

Without the two comments out code I will not have issue.
If with two comments out code, the complier will report that:
test.cpp:15:51: error: invalid use of incomplete type 'secp256k1_context_t {aka struct secp256k1_context_struct}'
 secp256k1_ecmult_context_clone(clone_ecm_ctx, &ctx->ecmult_ctx);
                                                   ^~
In file included from test.cpp:2:0:
/usr/local/include/secp256k1.h:50:16: note: forward declaration of 'secp256k1_context_t {aka struct secp256k1_context_struct}'
 typedef struct secp256k1_context_struct secp256k1_context_t;
                ^~~~~~~~~~~~~~~~~~~~~~~~

That looks like I can not operate the context struct internal member (because of the opaque struct), however, my code indeed need to access the struct inside of the context struct, for exp, the below code will trig a similar compiling error:
secp256k1_ecdsa_sig_verify(&ctx->ecmult_ctx, &r1, &s1, &q, &m)
is there any workaround there?

You use secp256k1_context_create () to initialize the context. Usually you will pass the flags SECP256K1_CONTEXT_SIGN or SECP256K1_CONTEXT_VERIFY

Can you add this as the answer @MarkH? I haven't verified it personally but I've been told it is correct.

There's an open PR to libsecp256k1 that adds example usage code https://github.com/bitcoin-core/secp256k1/pull/748 Perhaps that's helpful for you.

@MarkH Thank you so much for your suggestion, I add more into the questions.

@MarkH So far I still have the problem, using the posted small code can reproduce the issue I have, would you please help to check? Thanks so much...

Just an update of one solution, for whoever meet such an issue, that one can copy the definition of the   secp256k1_context_struct into the test.h, include this test.h in test.cpp, then you don't meet such compiler complain any more..

The reason for this error is that secp256k1_context is an opaque type. This means that you cannot create it on the stack; you need to create a heap-allocated version (and destroy it) using the functions from the API (specifically secp256k1_context_create and secp256k1_context_destroy, as Mark H mentioned in a comment). There is no way to access the internals.
You're right not to try to #include <secp256k1.c> ... aside from being unlikely to build and requiring you to bundle a copy of the whole library, it would expose internals of the library to your code which are not safe to use directly.
For what you are doing (replacing internal signing functionality with OCL-enabled versions) you are probably better off forking the library and directly editing the source. This isn't much more invasive than what you're trying to do, and it would let you use the existing unit tests and valgrind ctime tests to check your work.

first of all, Thank you all for the above comments and suggestions. I will add more into the original post.

secp256k1_context is an opaque data type. The secp256k1_context_struct is defined in secp256k1.c (not in secp256k1.h) which results in hiding internal data members from the callers.  Therefore, the caller can't access the context's internal data members (by dereferencing).
This implementation design is not limited to libsecp256k1. For example, file input and output in C uses the same method. This article will give a better understanding of implementing opaque objects in C.


text:
Bitcoins stuck in memory pool
I sent my 0.1 bitcoin on the 8th of January from an offline wallet (Bitcoin Core), to an online wallet. Still haven't got the btc in my new wallet. The status on the offline wallet says: "unconfirmed, in memory pool". When I paste the TX ID in the Blockchain it won't give any results. Has this happend to anyone else? If so, do you know how to fix it, or how long it takes before the BTC gets out of the memory pool?
Also, when I sent the 0.1 bitcoin, at first the offline wallet wouldn't let me send it because I didn't have enough transaction fee in my wallet. (I had 0.1 btc in my wallet) So I sent 0.09995520 btc which was exactly enough for my remaining btc to cover the transaction fees. So I don't think it's because of the transaction fees that my btc is now stuck in memory pool.

Does this answer your question? Why is my transaction not getting confirmed and what can I do about it?

Could you clarify what you mean with "offline wallet"? If the sending wallet is truly not connected to the internet and no other steps have been undertaken to submit the transaction to the Bitcoin network, I would have an outlandish theory why the Bitcoin network hasn't become aware of the transaction. ;)


text:
How to know Block height when a transaction entered Mempool?
I'm hosting a Listening Bitcoin Node in Capital of Pakistan. I want to know the Block height when Transaction entered the mempool. Right now I'm using the time from the Json format of the Transaction and Checking the Block height against it. Do you guys think that this method is reliable or I fear that transactions in my mempool have more latency than usual node in Europe. Should use an Api to find the Entry Block Height of the transaction?
Please also suggest an api also?

Bitcoin Core has a getmempoolentry command. This takes a txid and returns information about that transaction's mempool status. This includes height which is the block height when transaction entered pool.

Andrew I was talking about the historical data

@HaseebSaeed You should clarify that in your question. Historical data of this kind is not stored by Bitcoin Core. It isn't relevant or useful to node function so it isn't stored. But you could write a program which records this information for every transaction that enters the mempool. You could use the ZMQ interface to be notified of when a transaction enters the mempool and then use that notification to do getmempoolentry and save the information wherever you want.


text:
How to segregate wallet from bitcoin core | Run multiple wallets in different servers on one centralized bitcoin node
I am running bitcoin full node in a centralized server. I have many web applications running in various places. I want to connect this web applications with centralized full bitcoin node. How to install wallet in these web application server which will be connected to one centralized bitcoin full node. I want to generate address, make transactions and sign transaction independently in these web application server using wallet and send it to the application server. Application server need to communicate with wallet using rest apis. Finally this singed transaction should send to centralized bitcoin full node. How to do this?


text:
Unused Output Tree in the Blockchain
I have been reading about the security of thin clients and I came across the section that discusses the proposal of Unused Output Tree in the Block chain.

Unused Output Tree in the Block chain (UOT)
  There have been several proposals (the first appears to be this one by gmaxwell, who called it an "open transaction tree", although the term "open" is now taken to mean "not yet mined into the block chain" rather than "unspent") to form a tree of unused transaction outputs at each block in the chain, hash it as a Merkle tree, and encode the root hash in the block chain (probably as part of the coinbase input). This will be called an Unused Output Tree (UOT). The first detailed proposal so far appears to be Alberto Torres' proposal; etotheipi's ultimate block chain compression is a variant of this.
If such UOT hashes were included in the block chain, a client which shipped with a checkpoint block that had a UOT would only need to download blocks after the checkpoint. Moreover, once the client had downloaded those blocks and confirmed their UOTs, it could discard all but the most recent block containing a UOT.

I understand this part, and how the merkle tree is going to be constructed, but the next part that talks about associated risks is what confuses me:

Hostile miners may insert blocks into the chain which have what claims to be a UOT, but which is actually invalid. It is unlikely that such blocks could be kept out of the chain because, again, this would require adding a new block validity criterion, and miners implementing this new criterion would risk "mining on the wrong side" of a fork, which could cost them a lot of money. Therefore, any UOT strategy would need to cope with the fact that not every block containing a UOT entry can be trusted.

So my questions are:

I assume that an invalid UOT means a tree that contains outputs that are either double spend, or doesn't even exist in the chain. Why wouldn't the bitcoin network be able to reject those? Also are full nodes also vulnerable to this attack? I think that Full nodes can always check the outputs in the UOT against the previous to make sure that all outputs are either included in the prev UOT or generated by transactions in the current block.
I understand that implementing this would require some modifications to the current validity criteria, but whhy would nodes up-taking these changes risk "mining on the wrong side"?