text:
Question about transaction verification relating to the ECDSA "nonce"
Context about the ECDSA nonce: Questions about generating a random number for ECDSA
Are these nonce values public or private? I understand that they are generated to be different every transaction and never reused, but unless I'm mistaken I cannot find any reference to them being a public part of a bitcoin transaction. Because of this, I don't understand how a transaction could be verified if a secret nonce was used, would this not obscure a large part of the message being signed and create un-verifiable data from someone who doesn't own the private key? Or is it that the value is in fact public, if public where are they in a transaction? If it is private how are verifications accomplished?

Are these nonce values public or private?

Yes and no.
For every signature, a new, secret, unpredictable, private nonce is generated. It is never revealed, just like the private key is never revealed. If the term "ECDSA nonce" is used without qualifications, it usually refers to this private nonce.
Then the corresponding public nonce is computed. It's effectively the same computation as is done to go from private key to public key. This public nonce is part of the signature, and necessary to verify it. This is usually not talked about in superficial descriptions of ECDSA, as all of this is just internal to the algorithm. For users of ECDSA, it suffices to think of the signature as a single black box that ought to be passed around from the signer to the verifier unmodified.
I'll try to explain how ECDSA works in a bit more detail, though you have to be aware that there isn't all that much of an "insight" that makes one understand why it is secure; ultimately all of that boils down to many people having tried to break it, and fail.

Let m be the message, and z = hash(m) the hash of the message.
Let d be the private key, a number in range [1,n-1], where n is a constant, close to 2256.
Let E be the function that turns private keys into public keys, which takes as input a number, and returns a "point".

E has the property that for any a and b, it holds that E(a) ⊕ E(b) = E(a + b), where ⊕ is an addition-like operation between points.
E also has the property that for any a and b, it holds that aⓧE(b) = E(ab), where ⓧ is a multiplication-like operation between numbers and points, returning a point.
E has the property that given E(d), it's hard to find d.

Let Q be the public key, Q = E(d).

ECDSA signing of message m with private key d is then roughly:

Generate a random nonce k.
Compute the public nonce R = E(k).
Turn R into a number, r.
Compute s = k-1(z + dr).
Return the signature (r, s).

Someone seeing the signature can't figure out k or d, because it's hard to go from R = E(k) to k, and without k, the equation s = (z + dr) / k has two unknowns (k and d). Note that this is just an intuition for why it's hard; it's not a proof.
However, it is possible to verify the signature!

We know s = k-1(z + dr).
Bring k to the other side: sk = z + dr.
Apply E() to both sides of the equation: E(sk) = E(z + dr).
Then use the multiplication and addition rules: sⓧE(k) = E(z)⊕(rⓧE(d)).
Using R = E(k) and Q = E(d) we get: sⓧR = E(z)⊕(rⓧQ).
This is an equation the verifier can check, since they have s, z, r, Q, and can find R from r.

In actual ECDSA, the ⓧ and ⊕ operations are elliptic curve point multiplication and addition, and E() is multiplication with the curve generator. Those are implementation details however, and the scheme works just as well for other kinds of mathematical structures that permit such operations, and have the property that E() is hard to invert.

Is the private nonce (k) range same as the pvtkey range: a number in range [1,n-1], where n is a constant, close to 2^256 ?
We use the same function E() to generate the public nonce and the public ECDSA key. I am using the rust secp256k wrapper: secp256kRust in order to generate my public key from my private key. It expects something like the output of a sha256 hash for the private key  arg, is the private random integer value sha256'd before running E(k) to get the public one? If not how can I run the pubkey gen fn E() on it?

There's some functions that are not demonstrated in this crate that seem to reference RFC6979 signEcdsaRecoverable but the docs docs seem to use a function (sign_ecdsa) that does not use the nonce?

My explanation above is very abstract! This is not how practical implementations work. I just wanted to give an intuition for how it works, not going into details.

But to answer your question: the nonce k is in the same range as private keys, from 1 to n-1 inclusive (where n=0xfffffffffffffffffffffffffffffffebaaedce6af48a03bbfd25e8cd0364141); the nonce k is usually generated using RFC6979 (which effectively computes it as a function of the private key and the message). The private key is generally not a SHA256 hash, but it can be; many Bitcoin wallets these days use BIP32 for generating private keys.

The documentation for that crate seems outdated. None of the libsecp256k1 functions accept you passing in your own nonce; the nonce is always generated by the library, because it's so easy to screw up if you do it yourself.

Is this function low_r using the RFC6979 a you described: a function of the private key and the message? I am sort-of confused because I assume that if there is a random number being selected upon signing, then the signature output should change upon each signature, even if the message stays the same. Using the sign_ecdsa_low_r function produces the same output every time, which means it is not adding a random value right? Or is it producing the same deterministic value based on the pvt key and message?

k in practice is not random; it is merely unpredictable to anyone who does not know the private key. RFC6979 specifies determining k as a function of the private key and the message, which suffices - it'll be different for every message, and unpredictable without access to the private key.

This is very clear and helps me understand the underlying mechanism, however I'm still trying to apply this to confirm if the Rust crate is correctly using this k value? I understand that it would be a different k for every different message, but does that mean that the k value is the same for every identical message assuming same private key?

libsecp256k1, and by extension rust-secp256k1, deals with nonces internally. You don't need to do anything. Yes, without additional randomness, if the message and private key are identical, the nonce and hence the entire signature will be identical too.

If you have further questions about the rust-secp256k1 crate, or libsecp256k1, please open another question, as that all seems unrelated to the question being discussed here.


text:
Inconsistent behavior testing Taproot Workshop - Section: 0.2 Elliptic Curve Math
Workshop link: https://github.com/bitcoinops/taproot-workshop/
In the section 0.2.4 Programming Exercise: Distributivity of scalar operations  we implement the following code:
a_key = ECKey().set(a)

b = random.randrange(1, SECP256K1_ORDER)
b_key = ECKey().set(b)

c = random.randrange(1, SECP256K1_ORDER)
c_key = ECKey().set(c)

# Left: Compute a - b as ints (modulo the sepc256k1 group order)
a_minus_b =  (a - b) % SECP256K1_ORDER# TODO: implement

# Left: Compute (a - b) * c as ints (modulo the sepc256k1 group order)
left =  (a_minus_b * c) % SECP256K1_ORDER# TODO: implement

# Right: Compute a * c - b * c as ECKeys
right = (a * c % SECP256K1_ORDER) - (b * c % SECP256K1_ORDER) # TODO: implement
#if you dont modulo curve order in both parenthesis your number (probably) becomes too large for the curve
#therefore calling .secret on it will not work even if you cast it to ECKey Object (so the assertion cannot even happen in this case)
#you would only be able to call .secret on a value within the curve order

print("Left: {}".format(left))
print("Right: {}".format(right))

right = ECKey().set(right)
# Left/Right: Assert equality
assert left == right.secret
print("\nSuccess!")

Note that the lines with #TODO: implement are the only ones I have modified.
When trying this code block a few times I noticed that it fails occasionally with:
Left: 84229569338898829804715923445734053841060795723920762893503652295039608159004
Right: -31562519898417365618855061562953854011776768555154141489101510846478553335333
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[32], line 28
     26 right = ECKey().set(right)
     27 # Left/Right: Assert equality
---> 28 assert left == right.secret
     29 print("\nSuccess!")

AttributeError: 'ECKey' object has no attribute 'secret'

The attribute error suggests that the generated secret is outside the curve order and was not properly turned into the ECKey
But for at least 50% of the time it returns something like:
Left: 51082417157028894624564857296082907029625179491897309339882235219613900809295
Right: 51082417157028894624564857296082907029625179491897309339882235219613900809295

Success!

What is causing this inconsistency?

Your right is calculated with (a * c % SECP256K1_ORDER) - (b * c % SECP256K1_ORDER). There's no guarantee that one side of the - will always be bigger than the other side, so it's possible that (b * c % SECP256K1_ORDER) is bigger than (a * c % SECP256K1_ORDER) which results in right being a negative number. As a negative number, it cannot be a valid scalar, and so its secret is not set. Since all of the numbers are random, this happens with 50% probability.
To resolve this, you just need to handle the negatives correctly. A negative number in a prime is just its absolute value subtracted from the prime (SECP256K1_ORDER). How you achieve that is left as an exercise to the reader :)

That makes tons of sense thank you for the clear explanation!

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Why timestamp mandatory on importdescriptors?
Stemming from How can a private key be imported to a descriptor wallet? (comment), can't really understand how timestamp is a mandatory argument.
Every elements in a descriptor have an activation date. Like public keys from genesis and witness from segwit ca. 2016/2017. Propagating those up the descriptor format/lang, this causes the descriptor as a whole have a minimum sensible timestamp.
The current source code just seems to add unnecessary usage complexity, like... in cases when the user doesn't know what's the useful timestamp for some descriptor, they are forced to choose (potentially wrongly) an arbitrary date instead of letting the source code make an informed guess based on analyzing descriptor elements to infer a minimum reasonable guess, or analyze everything anyway as a most naive solution.
I know this reasoning isn't entirely valid because one can add arbitrary data before softfork activation, but even so, let's consider a reasonable timestamp is genesis for any possible descriptor.
How does this not make sense, for timestamp is mandatory in importdescriptors?

As in every open-source project, the answer to "Why isn't feature X implemented?" is simply that nobody has added it. Maybe whoever worked on it didn't consider the option, maybe they found it too much work, or maybe they didn't think it'd be valuable.
Even though the feature may appear obvious or even trivial to you as a user, it still requires developer and reviewer time to add it. Unless you pay someone to do it for you, you're not entitled to other people's time.
Pull requests welcome, of course.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Why Bitcoin design is orange?
Why is Bitcoin colored everywhere in orange even when the historic late 2000's color was some yellow-ish / gold-ish? In some places e.g. the Bitcoin group at GitHub, it seems to be portrayed showing a less informal black color.
Some additional related doubts, maybe for other independent posts:

Why did Core specifically choose orange for their color?
Why is the large blockers' fork so frequently portrayed in green color?
Why is the Wright fork so frequently portrayed in yellow color?
Why is testnet colored in green and signet colored in pale green (in Core)?

Why is Bitcoin colored everywhere in orange even when the historic late 2000's color was some yellow-ish / gold-ish? In some places e.g. the Bitcoin group at GitHub, it seems to be portrayed showing a less informal black color.

Orange is used everywhere as a consequence of Core having chose that color at some point. In a way, you could say it is a measurement of influence. Sometimes that's desired, sometimes not. Sometimes is conscious, sometimes not.
Really, any color or design can be used to refer the Bitcoin chain, the one that hasn't hard forked itself for a while. At your own risk: your customers could be confused by your choices.

Why is testnet colored in green and signet colored in pale green (in Core)?

For signet it looks like simply acknowledging the raw result of the HSV arithmetic that fitted the middle distance between orange and green.
Really, any color or design can be used to refer that signet chain. That explains why mempool.space deviates signet to purple, for whatever reason.

The current Bitcoin logo was adopted in the reference client's version 0.5.0 in 2011 (but it wasn't until 2014 that it rebranded to the current name Bitcoin Core). The logo was designed by bitcointalk user "bitboy" here: https://bitcointalk.org/index.php?topic=1631.msg19455#msg19455, and was not intended to be specific for one piece of sofrware.


text:
"pure" P2WSH invoice address in legacy wallets?
If I got it right, my legacy wallet seems to be able to get me:

2000s-era P2PKH invoice addresses (appearing as base58).
P2SH-wrapped P2WSH invoice addresses (appearing as base58).
P2WPKH invoice addresses (appearing as 42-length bech32 sequences).

If I got it right, it won't ever get me P2TR invoice addresses until migrated into descriptors wallet.
But why it doesn't feature a stream of "pure" P2WSH invoice addresses (appearing as 62-length bech32 sequences)? Is it because those make sense only in non-trivial script environments such as multi signature or others, and therefore hidden to me as a shorter-length-addresses YAGNI simplification?

If I got it right, my legacy wallet seems to be able to get me:

Classic P2PKH invoice addresses (appearing as base58).
P2SH-wrapped P2WSH invoice addresses (appearing as base58).
P2WPKH invoice addresses (appearing as 42-length bech32 sequences).

Almost. It is producing P2SH-wrapped P2WPKH; not P2SH-wrapped P2WSH.

If I got it right, it won't ever get me P2TR invoice addresses until migrated into descriptors wallet.

Only descriptor wallets support creating P2TR receiving addresses. It still requires actually having a taproot descriptor in that wallet. This won't become the default until probably some time after taproot is active on mainnet (currently estimated to happen around November 16th 2021).

But why it doesn't feature a stream of "pure" P2WSH invoice addresses (appearing as 62-length bech32 sequences)? Is it because those make sense only in non-trivial script environments such as multi signature or others, and therefore hidden to me as a shorter-length-addresses simplification?

Well, let's take a step back. P2WSH, like P2SH, is a mechanism for turning other scripts into addressable outputs. So what you're talking about is probably really something like P2WSH-P2PK or P2WSH-P2PKH. Both of these are completely pointless, as they're strictly more expensive to use than the corresponding P2WPKH variant. P2WSH is only useful when there is a script that's more complex than a single-key policy, like multisig, or the more advanced contracts used e.g. in Lightning.
It isn't so much that these are hidden from you - they're just not supported by the legacy wallet as automatically-created addresses. It is possible to explicitly import such a script (I think), and it would work, but you'd need to do it for every address individually. The wallet simply generates public keys, and can expose them as P2PKH, P2WPKH, or P2SH-wrapped P2WPKH addresses.
In a descriptor wallet, if you wanted to, you could import a descriptor for such a P2WSH-P2PK(H) script if you wanted to, and it'd actually generate addresses for it. If you really wanted to, you could even construct P2SH-P2WSH-P2PKH scripts. All of these are still pointless, though.

If P2SH (when used for wrapping) walls contents behind it in some way that contents aren't revealed until spent, then why getnewadress label p2sh-segwit defaults to P2SH-wrapped P2WPKH instead of optimizing it to P2SH-wrapped P2WPK? Is it because OP_HASH and the pubkey hash is cheaper than just the pubkey? (I can promote this to question if that's more comfortable).

There is no such thing as P2WPK; the only valid witness v0 programs are P2WPKH (OP_0 + hash160 of pubkey) or P2WSH (OP_0 + sha256 of script).


text:
Will "all" bitcoins ultimately vanish? (be unusable)
During this CCC conference from 2011, starting from 00:18:55, the presenter suggests that as bitcoins can be lost (lost private key, dead owner, sending to an "eater" address that nobody possesses, etc.), and as the total amount of bitcoins has a known threshold (21 millions), there is a possibility that given enough time, most bitcoins, if not all, will actually vanish (be unusable).
The questions is when?
Actually, according to him:

if 0.5% of bitcoins are getting lost per year, "all" bitcoins will be gone by approx. 2032;
if the losing rate is 0.1% per year, it will probably last for a century, but will ultimately be doomed.

And if we suppose that inactive wallets (2 years with no transaction) are in fact lost wallets, we reach that 0.1%/year threshold...

It could happen but I'd say it's highly unlikely. Perhaps as time approaches infinity, number of bitcoins approaches 0.

related: http://bitcoin.stackexchange.com/q/484/5406  http://bitcoin.stackexchange.com/q/9923/5406

possible duplicate of What could be the consequences of many bitcoins being lost out of circulation if people lose their wallets?

Talking about the loss rate as any percentage of total bitcoins is nonsense. It should be the percentage of wallets that are lost per time unit. As bitcoins gain use and value surely the number of wallets will increase with smaller and smaller amounts in each.

Just watched the video: Kay Hamacher's scenario assumes that 5% per year disappear. His chart shows an absolute decline, i.e. 5% of the total amount of Bitcoins that ever existed are lost per year. (Which is absurd for the reasons discussed below).

The math is clearly absurdly unrealistic because as there are fewer Bitcoins available, it becomes harder to lose them. So the rate of loss has to decay as the availability of Bitcoins goes down.

Although the quantity lost per iteration will fall, as the total supply decreases the RATE of loss be it half a percent, one percent, etc. per year would not change, as the circumstances leading to loss would be unchanged; death, drive failure, natural disaster, etc. The rate of loss will always be n^T where n is less than one; ie 0.99 and T is time.  It would seem to me that it is you that fails to understand logarithmic decay; half-life.

Are you suggesting that 0.5% relative to the still available bitcoins are lost every year or 0.5% of the absolute amount of bitcoins? Anyway, either my math is lying or each would turn out to be about 10% loss in 20 years, which doesn't translate to "all" in my opinion.
Additionally, should the value of Bitcoin sufficiently increase (which would happen if as many Bitcoins were lost as you suggest), the protocol could be hard-forked to allow for another eight digits divisibility. While the distribution would remain the same, a fraction of a bitcoin would then still be enough to fuel a global economy.
Finally, since this question was first asked, wallets with deterministic key derivation have become more commonplace. As it is sufficient to transfer the master seed for such wallets, it is almost trivial to bequeath such wallets.

Excelent analysis! A loss of 0.5% of each year's existing money supply means that, over 20 years we get 1.005^20 = 1.1049 i.e. 10.49%  If it's rather 0.5% of all possible Bitcoins (that is, off the max. possible 21 million) then in 20 years we get exactly 10% . Per your 2nd definition, bitcoins would evaporate in 200 years, as per the first, in ln(2)/ln(1.005) = 138.9757 years :)

Thanks! However, in the relative loss scenario, the loss should be 1 - 0.995^20 = 9.54%. Otherwise one would calculate a 0.5%/a growth, not a 0.5%/a loss.

There is currently a bounty offered on BitcoinTalk for implementing a method that would add bitcoins back to the mining reward in the case that an address had no activity for over 50 years which would completely resolve this concern.
The post can be found here and is located at #3 on the list https://bitcointalk.org/index.php?topic=371601.0

The math associated with this question is simply wrong! Conceptually, it is very simple to apply the rule of 72 in reverse. By applying the rule of 72, known by most bankers,  knowledgeable investors, and knowledgeable consumers of loans, it indicates half of all Bitcoins will disappear after 144 years, assuming 0.5% are lost each year. (No PHD in mathematics is required, very simple math: 144 = 72/0.5)

Every year 1% of people die.
If every year 1% of bitcoin is lost, every 70 years:
HALF the coins in circulation remain.
if all coins were accounted for and all were already mined:
2086 0.99^70  = 50% remain
2156 0.99^140 = 25% remain
2226 0.99^210 = 12.5% remain
2296 0.99^280 = 6.25% remain

-1 This calculation makes weird assumptions because it seems extremely unlikely that 1% of the population dying that translates to 1% of the Bitcoin users dying. Also, 1% of the Bitcoin users dying doesn't necessarily translate to 1% of the Bitcoins lost. Thus, I'm not sure what we're supposed to learn from this answer.

so you're proposing that even though 1% of all people die every year... bitcoin holders have found the fountain of youth and therefore 1% of bitcoin holders don't die like normal people?

Further, besides death and failure to will; there is house fire and paper wallets, and coins lost on hard drives that fail every 10 years unlike humans which tend to make it 70+ years, then there are coins lost because sent to invalid addy, and myriad of other failures.  I suspect 1% annual loss of supply is conservative.  http://i.imgur.com/6C238Pp.png

No, I merely suggest that since Bitcoin is a young technology, it is adopted first mostly by 20-40 year olds, and that those age groups have a smaller portion of the dying population for decades to come. Also it seems rather absurd that with gaining importance and value people would continue to lose Bitcoin at the same rate, especially since backups of deterministic wallets (that have become more common since this question was asked) don't outdate and are thus are perfectly inheritable.

In theory, yes, they will all disappear eventually. However, as the money supply decreases due to their being less Bitcoin in circulation, the price will increase due to the scarcity of the existing coins so it will become more and mor expensive to misplace your bitcoins :)

-1 Not useful to me: Repeats information from the previous answers and doesn't add any new insights.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
With the ongoing War, do you think it will affect Bitcoin Mining?
The Russia-Ukraine war has so many effects on inflation in many countries. Will it also affect the mining of Bitcoin, especially cloud mining?

The network hash rate has continued on an upward trend throughout 2022 and doesn't seem to have been impacted at all by the Russia-Ukraine war. It is plausible that mining farms in the region could have been forced offline due to lack of electricity or lack of connectivity but without drilling into the numbers I can't tell whether this has happened or not. I suspect the vast majority of the network hash rate isn't located in the affected region but again I don't have access to any particular data to back that up.

thank you for answering, so there is no effect on cloud mining companies such as moden, nicehash, bitfeeder and etc.?

@vash: It depends where the servers are located and if those servers maintain uptime. I don't have the data on either, but at least at a network level it doesn't seem to have had much/any impact.

alright, thank you for answering my questions.


text:
Is tweak used when spending from script path for Taproot?
I understand that a taproot output can be either spent using key path or script path.
With key path, a tweak is applied to the private key and then sign the sighash. The signature is provided as a witness.
However with script path, I do not see the tweak being used anywhere. For example, in 2.4.8 in
https://github.com/bitcoinops/taproot-workshop/blob/master/solutions/2.4-taptree-solutions.ipynb, the privkey_a is used to sign the sighash to use TapScript_A, but nothing involves a tweak when constructing the witness [signatureA, TapLeafA.script, control_map[TapLeafA.script]].
My question is, if I want to construct a taproot output just with script spend, what's the point of using tweak?

Related question: https://bitcoin.stackexchange.com/questions/99722/taproot-eliminating-key-path

You perform tweaking at signing time if the key you're signing for was tweaked. That's a decision that was made when the output was constructed, not when it is spent.
Taproot outputs can consist of:

Just a raw public key directly, with no tweaking. In this case only a key path spend is available, by signing with the corresponding private key. It's just a pay-to-pubkey now, and there is no tweaking at any point. This approach is not advisable for reasons I won't go into, but it is perfectly legal.
Just an internal public key, with no script tree, which is tweaked to construct the output key using H_TapTweak(pubkey) as tweak. In this case again only a key path spend is available, but at spending time the private key needs to be tweaked by the same amount or it wouldn't match the published public key. This is what most single-party taproot wallets use.
The full construction with an internal public key, and a tree of one or more scripts. The output now consists of the internal key tweaked using H_TapTweak(internal_pubkey || merkleroot), where merkleroot is the Merkle root of the script tree. Such outputs can be spent in one of two ways:

The key path spend, which involves signing with the tweaked private key, corresponding to the tweaked public key that was published in the output.
A script path spend, where one of the script leaves, the internal (untweaked) key, and a Merkle path proving that script occurred in the tree, are revealed, together with whatever that script needs to satisfy it. If that script involves CHECKSIG operations, signatures must be provided for the public keys in the script. If those public keys are tweaked, the signin step will need to tweak the corresponding private keys too, but this is not necessary or advisable. Typical constructions use untweaked keys inside taproot scripts.

So to summarize: you need to tweak private keys at spending time when the public key you're signing for was tweaked. Whether that is the case depends on how the output was constructed. The internal public key (the one intended for key path spends) is usually tweaked (and must be, if scripts are present). The public keys inside those scripts usually aren't tweaked (but in theory they could be).

the privkey_a is used to sign the sighash to use TapScript_A, but nothing involves a tweak when constructing the witness [signatureA, TapLeafA.script, control_map[TapLeafA.script]].

The control block (taken from the control_map in the code you cite) contains the tweak that was used. It's just not used to sign the private keys, because the public keys in the script were not tweaked either. But the tweak must still be revealed, or verifiers cannot tell that the script you're revealed actually contributed to the published public key in the output.

My question is, if I want to construct a taproot output just with script spend, what's the point of using tweak?

You can't, really. Every taproot output that permits script path spending is constructed by taking an internal public key and tweaking it with the Merkle root, and you must know what that internal public key and tweak are to spend it, as you need to convince the verifier "wait! that public key you saw in the output, it's actually tweaked using my script here, so I can spend with that instead!". If you don't want the ability to spend with any internal key using a key path, you'll need to use an internal key that nobody knows the private key to; but there still must be one.
The philosophy behind Taproot is that nearly always, outputs can have an internal public key corresponding to some or all of the participants' consent, and heavily optimizes for being able to sign using the key path as a result. If you have a use case with just a script, the result may feel unnatural due to the need for a dummy internal public key, and tweak.

Awesome, thank for clarifying.

The taptweak is defined as t = TAGGED_HASH(P | s) where P is the internal public key and s is the Merkle Root with all tapleaves. You provide both P and s, so the tweak is computed using the mentioned formula. To get the external public key (the one in the spk) you just compute P + tG


text:
How to connect Sparrow wallet to Bitcoin Core on a different machine but the same network
I have a Raspiblitz set up on a Pi 4 and am trying to connect Sparrow wallet to Bitcoin Core from another machine on the same network. I have read through the instructions at sparrow.com/docs but cannot get Sparrow to connect.
Looking at my bitcoin.conf file on the Raspiblitz the layout is slightly different to the layout in the example:
Example Layout:
rpcuser=bitcoin
rpcpassword=yourpassword
[main]
rpcbind=127.0.0.1
rpcbind=192.168.0.10
rpcallowip=127.0.0.1
rpcallowip=192.168.0.0/16

My layout
rpcuser=bitcoin
rpcpassword=yourpassword
main.rpcport=8332
rpcallowip=127.0.0.1
main.rpcbind=127.0.0.1:8332

I have inserted my specific IP address (192.168.1.10) by adding the lines:
rpcallowip=192.168.1.0/16
rpcbind=192.168.1.10

In that order underneath like in the example. I don't know much about the CIDR notation at the end of the rpcallowip address but have also tried 192.168.1.0/24 and the address itself without the / (192.168.1.10). I've also tried main. in front of the rcpbind for my IP.
Can someone point out what I am doing wrong. I have looked for answers but there are not many too many guides for this aspect of Sparrow.
Thanks

Is it possible you have a firewall blocking the port? try adding a rule to allow connections on the rpc port (8332 as you specified)


text:
What will be the security risks if we persist "e" as part of schnorr signature=(r,s,e) and extract publicKey from signature at verification
We know unlike ecrecover, it is impossible to get to public key given schnorr signature = (r,s) since rG = sG-eP .
So, would like to understand what are the security risks,
if we say
signature=F(r,s,e)
and get to publicKey at the verification? something like this
P = (rG-sG)/e

What is F() ?

I meant concatenation @PieterWuille

There are no security risks to this since e is public anyway. Depending on how you are using it you probably want to make sure that e is consistent with R afterwards i.e. recompute e from P and R and m to make sure it is correct. However, it makes more sense to me to just include P directly instead of e.

Okay, thank you @LLFourn. But my point/question is to derive publicKey from signature.

But it's the same amount of data to just include P. Why can't extracting the trailing 32 bytes count as deriving?


text:
What’s going on with this "NULL address input" transaction?
I was reading this from the Bitcoin wiki (https://en.bitcoin.it/wiki/From_address). I understand the concept of UTXOs and that any input to a transaction is an output from a previous transaction. I also have a rough understanding of how ScriptSig and ScriptPubKey interact. The article mentions "advanced transactions will probably defeat most block explorers" and gives examples of some transactions one of them being:
54fabd73f1d20c980a0686bf0035078e07f69c58437e4d586fb29aa0bee9814f
When looking at this transaction in say blockchain.com the input is marked as "unknown" and when looking at the JSON for the transaction the input address is "null"
  "inputs": [
    {
      "coinbase": false,
      "txid": "a60143eb3f8d3cd1f42cca874f35736186d67c488efd3c1b7214bbd74b310e0c",
      "output": 0,
      "sigscript": "483045022100d92e4b61452d91a473a43cde4b469a472467c0ba0cbd5ebba0834e4f4762810402204802b76b7783db57ac1f61d2992799810e173e91055938750815b6d8a675902e014f",
      "sequence": 4294967295,
      "pkscript": "76009f69905160a56b210378d430274f8c5ec1321338151e9f27f4c676a008bdf8638d07c0b6be9ab35c71ad6c",
      "value": 9000000,
      "address": null,
      "witness": []
    }
  ],

I then looked at the input TXID a60143eb3f8d3cd1f42cca874f35736186d67c488efd3c1b7214bbd74b310e0c and the output address is also "null"
  "outputs": [
    {
      "address": null,
      "pkscript": "76009f69905160a56b210378d430274f8c5ec1321338151e9f27f4c676a008bdf8638d07c0b6be9ab35c71ad6c",
      "value": 9000000,
      "spent": true,
      "spender": {
        "txid": "54fabd73f1d20c980a0686bf0035078e07f69c58437e4d586fb29aa0bee9814f",
        "input": 0
      }
    }
  ]

Can someone explain for me what is going on here? And how one goes about creating these "advanced transactions" that apparently defeat block explorers that the article mentions (as in what makes them special to break block explorers?)
I have seen the question at Can PUSHDATA/OP_RETURN be used to break a trail of transactions permanently? and the answer talking about the "nonstandard" vout type but I don't understand what this means.
EDIT: I have seen "null" addresses in the context of OP_RETURN as outputs but my understanding is that those can't ever be spent so I don't understand how they are used as inputs.

When a receiver asks to be sent money, they specify the conditions under which they want to be able to spend the funds in an output script. Later when the receiver wants to spend their funds, they need to provide an input script that satisfies the output script of the output they are spending. In transaction validation, the input script is evaluated first, then the resulting stack is used as the starting point to evaluate the output script.
For example with P2PKH, the input script contains a signature and a public key, the output script contains OP_DUP OP_HASH160 pubkeyhash OP_EQUALVERIFY OP_CHECKSIG.
In evaluation the input script pushes first the signature then the pubkey on the stack. The stack is then passed to the output script which:

duplicates the pubkey
replaces the first of the two pubkey copies with a hash of the pubkey
pushes the pubkeyhash to the stack
Verifies that the pubkeyhash pushed from the output script and the pubkeyhash hashed from the pubkey in the input are equal
Checks that the remaining pubkey and signature amount to a valid signature of the transaction.

There are a number of standardized output script templates that cover the most common uses. Some of these cover single-sig usecases, but there are also multiple standard output types for complex scripts. Addresses are a convenient shorthand to communicate the receiver’s output scripts to the sender for standard output script types.
Even before P2SH was introduced, a receiver could define arbitrary conditions by writing out the corresponding output script using the opcodes defined in Bitcoin Script. These bare scripts are uncommon, since their arbitrary content does not lend itself to an address standard. The UX is horrible: instead of an address with a checksum, the receiver and sender have to exchange the actual script, and the sender needs to create a raw transaction manually specifying the output script. P2SH was introduced to improve the UX around defining your own spending conditions while allowing for an address standard.
The transaction you are looking at contains such a bare script: instead of following one of the standard output schemes, the receiver defined their own output script and satisfied it accordingly in the succeeding input.
The output script specified in the output a601…0e0c:0 of the preceding transaction is:
OP_DUP
OP_0
OP_LESSTHAN
OP_VERIFY
OP_ABS
OP_PUSHNUM_1
OP_PUSHNUM_16
OP_WITHIN
OP_TOALTSTACK
OP_PUSHBYTES_33 0378d430274f8c5ec1321338151e9f27f4c676a008bdf8638d07c0b6be9ab35c71
OP_CHECKSIGVERIFY
OP_FROMALTSTACK

The input script in the first input of 54fa…814f is:
OP_PUSHBYTES_72
3045022100d92e4b61452d91a473a43cde4b469a472467c0ba0cbd5ebba0834e4f4762810402204802b76b7783db57ac1f61d2992799810e173e91055938750815b6d8a675902e01
OP_PUSHNUM_NEG1

The script essentially amounts to an obfuscated version of a P2PK output as can be seen by evaluating the script execution:

The input script pushes a signature onto the stack.
Current Stack (left is bottom): SIG
The number -1 is pushed onto the stack
Stack: SIG -1
The stack is passed to output script validation
The number -1 is duplicated
Stack: SIG -1 -1
A 0 is pushed onto the stack
Stack: SIG -1 -1 0
OP_LESSTHAN consumes two items (a, b) from the stack returns a 1 to the stack because a (-1) is less than b (0).
Stack: SIG -1 1
OP_VERIFY consumes the 1 on top of the stack and succeeds
Stack: SIG -1
OP_ABS replaces the top stack item with its absolute value
Stack: SIG 1
A 1 is pushed to the stack
Stack: SIG 1 1
A 16 is pushed to the stack
Stack: SIG 1 1 16
OP_WITHIN consumes three values (x min max) and returns a 1 because x is greater than or equal to the minimum and smaller than the maximum
Stack: SIG 1
OP_TOALTSTACK removes the top element from the stack and puts it on the alternative stack.
Stack: SIG, Altstack: 1
A pubkey is pushed on the stack:
Stack: SIG PUBKEY, Altstack: 1
OP_CHECKSIGVERIFY consumes the signature and pubkey and verifies that the signature is valid in the context of the transaction and pubkey.
Stack: <empty>, Altstack: 1
OP_FROMALTSTACK removes the top value of the alt stack and places it on the stack:
Stack: 1, Altstack: <empty>
The script succeeds because it ends with a single truthy value 1 on the stack.

These transactions may break some block explorers in the sense that some block explorers may only have support for standard scripts and would not properly display bare outputs. It seems to me that modern block explorers no longer suffer from that:
e.g. mempool.space shows the output script in the preceding transaction…

… and the spending transaction just fine.
In case "breaking block explorers" was understood as a privacy benefit, this transaction is not more private. In Bitcoin, transactions do not spend funds from addresses: addresses merely specify the conditions under which funds can be spent, but each input must specify exactly which transaction output it is spending.
The preceding transaction a601…0e0c created a single output a601…0e0c:0 with the mentioned bare output script that could be spent by the owner of that script, and the first input of 54fa…814f explicitly spent that a601…0e0c:0, to create another transaction output 54fa…814f:0 that can be spent by the receiver in control of the address 1GMaxweLLbo8mdXvnnC19Wt2wigiYUKgEB.
I.e. every UTXO is uniquely identifiable and the transaction graph is public information. The absence of an address has no privacy benefit.

Can you please elaborate a bit more on the first part? I don't quite understand what you mean. I understand the coin flow is from 1HHRAfK7sS9QVNsmYqfEHy4sY7dfJDXMT1 to 1GMaxweLLbo8mdXvnnC19Wt2wigiYUKgEB but what accounts for/what is the "unknown" in the middle?

I added the actual output script and input script. Does that answer your question or could you rephrase what you are looking for?

Sorry I posted a comment, then I thought I got it, then I didn't and so posted and deleted a bunch of comments. So I've learned these are called non-standard P2SH addresses. Can you elaborate on why this particular script creates a non-standard address as opposed to a 3 address?
Also I find it rather odd that the input JSON I put in my original question has a pkscript as part of it. I thought pkscripts were only present in outputs. What exactly does it do here?

This is not a P2SH address, it’s a bare output. The input lists the prior transaction’s output as a convenience because both the input and output script are required to validate an input. I think you were looking at the input of a601…0e0c though, rather than the input of 54Fa…814f when you saw the P2KPH output script of the transaction output it was spending—I updated my answer to provide a bit more context.

This is brilliant, I wish I could upvote your answer again. This has explained it extremely well. Slightly unrelated but is there something which helps convert the JSON pkscripts/sigscripts into OP code format? Doing it manually is extremely time consuming (for me at least).

@user2334659 The Bitcoin Core decodescript RPC can decode scripts given in hex to opcode format.


text:
Lightning: Can a channel partner betray with an arbitrary balance?
I know, that I can settle old balances in a closing transaction. But can I also settle arbitrary "fantasy" balances in a force close when my channel partner is long dead. By example by modifying the transaction DB of my lnd node?

No. You can pick the revoked state (or "old balance" in your terminology) that is most beneficial to you i.e. the highest old balance that you previously had in the channel and broadcast that one if you are seeking to cheat your counterparty. It may be that one of the previously revoked states is one where you own effectively all the balance in the channel (e.g. if you single funded the channel in the first place). But you can't create a state that didn't previously exist where you own all the Bitcoin in the channel without getting a signature from the counterparty in the channel. If the counterparty in the channel broadcasts a revoked state then you can under the current LN penalty mechanism seize all the funds in the channel with a justice transaction but again it requires a previous action from the counterparty.

@RenePickhardt: Circular rebalances would still require the channel counterparty to agree (and sign for) every change in state in the channel right?

Sry misread the question. Deleted my comment!


text:
Hardware wallet interface
Why would you use the hardware wallet interface as opposed to the software that comes with the hardware wallet, such as ledger live or trezor suite? Is HWI better in some way?

HWI allows you to connect your hardware wallet to your own Bitcoin Core full node. If you use the software that comes with the hardware wallet exclusively you are relying on a full node that the hardware wallet manufacturer is running and not your own. This may be satisfactory to you depending on your circumstances but it is certainly not as trust minimized as running and relying on your own full node.

Couldn't someone hack the HWI github repo and place malware in it, the same as the proprietary software?

@EoinMcQuinn: They could hack the maintainer (achow101)'s laptop and push malicious changes and they could in theory hack Microsoft/GitHub so the code you downloaded contained malicious code. We are talking high levels of paranoia and speculation here on next steps.

I suspect malicious firmware updates on the device would be the thing to worry about over malicious code in HWI repo but again highly speculative.

Can you do firmware updates via HWI?

@EoinMcQuinn: No, firmware updates are run on your hardware wallet. HWI is code running on your Bitcoin Core full node server. Firmware updates aren't pushed to your full node or to HWI

So does the hardware wallet require any firmware in order for HWI to function?


text:
How to get rid of 'ghost' wallets in Bitcoin Core?
I created few wallets with Bitcoin Core using file-> Create Wallet. It created directories for each wallet name. Only that was for experimenting so I deleted those directories.
The problem is that now when I run Bitcoin Core, it tries (and fails) to load each of those nonexistent wallets. For instance, one of those wallet names was wall2, so when I run bitcoin core now I get this message:
"Warning: Skipping -wallet path that doesn't exist. Failed to load database path'E:\bitcoin\wall2. Path does not exist."
And for every such message, I need to click Enter.
How can I make bitcoin core 'forget' and stop trying to load all those ghost wallets?

In the data directory, there should be a file named settings.json. Just delete the lines with the names of the wallets that you have removed.
The file needs to contain valid json, so you also need to make sure that the last name entry in the wallet array does not contain a trailing comma.
For example, if you created wallets named wallet1, wallet2, wallet3, and wallet4, the file should look like
{
    "wallet": [
     "wallet1",
     "wallet2",
     "wallet3",
     "wallet4"
    ]
   }

If you have deleted wallet2 and wallet4, then you need to change the file to be:
{
    "wallet": [
     "wallet1",
     "wallet3"
    ]
   }

Great! Thanks @Andrew Chow

@Andrew Chow is there a way to officially delete a wallet rather than this manual procedure?

@mikew No, there is not. Generally we avoid giving users an easy way to delete things that are necessary to spend Bitcoin as that can be a very big footgun.


text:
How to show the Op codes of scriptPubKey?
I've been learning about bitcoin and creating raw transactions. In the example I saw in a video when they use the command decoderawtransaction it outputs:
...
 "scriptPubKey": {
        "asm": "OP_HASH160 f20fe211102535e3c37bb0e659099387ddc035b5 OP_EQUAL",
        "hex": "a914f20fe211102535e3c37bb0e659099387ddc035b587",
...

But when I do it, it outputs:
...
 "scriptPubKey": {
        "asm": "0 b9ae3f60f9d4f5f22f81ae36803ff71f0d9a1b0d",
        "hex": "0014b9ae3f60f9d4f5f22f81ae36803ff71f0d9a1b0d",
...

How do I make it show the Op Codes?

There is nothing special that you need to do, decoderawtransaction will always show the opcodes. In fact, what you are looking at are the opcodes for that specific output. Not all outputs are the same, and so there will be different opcodes in use.
If your first example, the scriptPubKey contains OP_HASH160 and OP_EQUAL because it is a P2SH output. P2SH outputs have those two opcodes, in addition to the hash.
In your second example, the scriptPubKey does not contain any non-push opcodes because it is a P2WPKH output. P2WPKH outputs do not use non-push opcodes (it has push opcodes, but decoderawtransaction will hide them because the useful part is the data being pushed). So there are no opcodes to show in your second example because that is how the script is.


text:
How does Lightning Node communicate with Bitcoin node?
When the channel is closed between Lightning Nodes, how is the closing transaction sent to the Bitcoin network? Does that mean to run a Lightning Node, you also have to run a Bitcoin node on your local machine since you have to broadcast closing transactions in the end? If there is a remote way for Lightning Node to communicate with Bitcoin Node, does that mean Lightning Node also has capability to communicate under Bitcoin protocol?

Lightning Channels are just unbroadcast cached Bitcoin transactions. So yes, Lightning Nodes need to keep abreast of transactions on the Bitcoin blockchain to notice new channels getting opened, existing channels getting closed, and need to be able to broadcast Bitcoin transactions to close channels or to submit justice transactions.
Different Lightning implementations use different ways to interact with the Bitcoin network. Some require access to a full node of varying flavor (e.g. Bitcoin Core, btcd), others use a light-client model based on compact block filters.

Thanks for the answer! Just to clarify, to run Lightning Node, you need some kind of Bitcoin client running on your machine, correct?

This depends on the flavor of Lightning implementation you are running, but generally yes.

Do you have any recommendation to run lightning node without running full node, or run on light machine? I want to run node that is customizable so I want to avoid using hosted wallet service, but couldn't find any good option.


text:
Are multipart payments more expensive than normal payments? (Lightning Network)
I was reading the "Mastering The Lightning Network" book when I came across this part about Multipart payments which mean dividing a payment and sending it through multiple paths to increase the chances of the payment succeeding.
My question is if a payment is going to be sent through multiple nodes, isn't that going to increase the routing fees? If yes, wouldn't it be cheaper to send that transaction through the standard bitcoin network and not through Lightning Network?

Both onchain fees and Lightning routing fees are subject to market forces i.e. supply and demand. A key difference between onchain fees and Lightning routing fees is that onchain fees are driven by the size of the transaction rather than the amount being sent while Lightning routing fees are driven by the amount being sent. Large amounts sent through the Lightning Network either require large channel capacities, the use of multiple paths or both.

My question is if a payment is going to be sent through multiple nodes, isn't that going to increase the routing fees?

Possibly but not necessarily. It depends on the market forces. If there is lots of demand for using channels with large capacities they can charge a higher fee and then using a number of channels with smaller capacities may end up being cheaper. Similarly if there is little demand for routing payments across the whole network those channels with large capacities may charge a very low fee similar to the channels with small capacities and it may work out cheaper to use the channels with large capacities.
It is worth noting that there are currently two routing fees on the Lightning Network: a base fee and a proportional fee. If the proportional fee is the same across channels with large capacities as channels with small capacities then the total proportional fee will be exactly the same whether you use one path or multiple paths (assuming the number of hops is the same on each path). To work out which is cheaper would depend on a comparison of the base fees.

If yes, wouldn't it be cheaper to send that transaction through the standard bitcoin network and not through Lightning Network?

Again possibly but not necessarily. If it was a huge amount, say 10,000 Bitcoin, it would be cheaper to send that transaction onchain versus via the Lightning Network as today you probably wouldn't be able to send it through the Lightning Network and it successfully reach its destination (without channels being opened for the purpose of routing this particular payment). But you'd expect sending smaller amounts to be cheaper using the Lightning Network versus onchain regardless of whether that be using a single path or multiple paths.


text:
JSON-RPC signrawtransaction returns false when using private keys
I've been working with BitcoinLib testing various parameters on the signrawtransaction rpc call.
It's been mostly successful, however there seems to be a persisting problem when using only private keys to sign a transaction. (second optional argument of signrawtransaction call). The server replies false and returns the failed hex. I have tested the same transaction on the qt console using the same keys, and it returns true.
The wrapper employs a SignRawTransactionRequest class matching the four arguments of the rpc call.
public SignRawTransactionRequest(String rawTransactionHex)
    {
        RawTransactionHex = rawTransactionHex;
        Inputs = new List<SignRawTransactionInput>();
        PrivateKeys = new List<String>();
        SigHashType = SignRawTransaction.SigHashType.All;
    }

It allows for 'inputs' and 'privatekeys' to be null, following the signrawtransaction syntax, while assuming 'sighashtype=all' unless explicitly specified otherwise.
The implementation also includes a SignRawTransactionInput class corresponding to the first optional argument of the rpc call. ([{"txid":txid,"vout":n, "scriptPubKey":hex, "redeemScript":hex } ,...])
public class SignRawTransactionInput
{
    [JsonProperty(PropertyName = "txid", Order = 0)]
    public String TransactionId { get; set; }

    [JsonProperty(PropertyName = "vout", Order = 1)]
    public Int32 Output { get; set; }

    [JsonProperty(PropertyName = "scriptPubKey", Order = 2)]
    public String ScriptPubKey { get; set; }

    [JsonProperty(PropertyName = "redeemScript", Order = 3)]
    public String RedeemScript { get; set; }
}

The request if fed to the wrapper method RpcServer.SignRawTransaction and returns a SignRawTransactionResponse containing the result and the hex as returned by the server.
public class SignRawTransactionResponse
    {
        public String Hex { get; set; }
        public Boolean Complete { get; set; }
    }

So far, signings using just the hex (no other arguments), inputs, different sighashtypes and various combinations of those have been successful. The server replies TRUE and returns the hex of the signed transaction when called with plain hex

or when called with some arguments

However, it returns false when the second optional argument(list of private keys) is used. It has been tested with one, all, and various combinations of the available private keys of the wallet.

I suspect it might be a JSON error and that the wrapper is not constructing that part of the JSON properly, however that usually leads to a parsing error and not the false reply I get here.
The same combinations return true when signrawtransaction is called from the console.

Any ideas on what might be wrong with the private keys argument?

This was not a case of BitcoinLib malfunctioning in any way. It was simply a case of using wrong privkeys to sign the unspent outputs.

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
What does this functional test include?
Does the test test/functional/test_runner.py include all possible functional tests, or is it just a dummy initial test run?

You can run functional tests individually or you can run the test runner which will run all the functional tests.
From Jon Atack's doc on how to compile Bitcoin Core and run the tests:

test/functional/test_runner.py to run the standard test suite (try test/functional/test_runner.py -j 60 or a similar high number to run the tests more quickly in parallel)
test/functional/.py to run an individual test file
test/functional/test_runner.py --extended to run the extended test suite
test/functional/test_runner.py --help to see the various options for running tests

When should functional tests be run? Is it after changing a particular piece of code, or is always a good time to run functional tests?


text:
Difference and resources for Miniscript and Simplicity?
I have read about that it is a language for writing Bitcoin Scripts in a structured way, enabling analysis, composition, generic signing and more.
http://bitcoin.sipa.be/miniscript/ is helpful but need more information, examples, libraries etc.
Simplicity is a blockchain programming language designed as an alternative to Bitcoin script and works for Elements.
I need more links or information to try things.
Also can miniscript be used for elements and simplicity be used for bitcoin?

Miniscript is an encoding of Bitcoin Script. The functionality of Miniscript is a subset of Script so there is nothing you can do with Miniscript that you can't do with Script. Therefore scripts decoded from Miniscript can be used on Bitcoin mainnet today in addition to testnet, regtest, signet and Elements.
In terms of Miniscript resources (in addition to the one you posted), the following are useful:

Andrew Poelstra presentation at London Bitcoin Devs in 2020
Andrew Poelstra workshop at Advancing Bitcoin 2020
Blockstream blog post in September 2019
PR to add Miniscript support to Bitcoin Core (C++)
PR to add Miniscript support to Bitcoin Core test framework (Python)
rust-miniscript library (Rust)

Simplicity, depending on which jets are enabled, has greater functionality than Script and is a direct alternative to Script rather than just an encoding. In addition:

Simplicity is a work-in-progress low-level programming language with greater flexibility and expressiveness than Bitcoin Script. It allows you to verify the safety, security and costs of a program.

It is supported on this test branch of Bitcoin Core and in Elements but not yet on Bitcoin mainnet.
In terms of Simplicity resources, the following are useful:

Simplicity white paper.
Russell O'Connor presented on Simplicity in January 2018.
Adam Back presented on Simplicity in April 2020.
Blockstream posted blog posts in November 2018 and April 2020.
There is a Bitcoin Optech topics page on Simplicity.

There is also a Simplicity Discussions page in the Simplicity repo which has a number of interesting Q&A. A more technical question on Simplicity may have already been asked there.


text:
Why exactly would adding further divisibility to bitcoin require a hard fork?
There are quite a few threads already on this topic, but they all focus on the necessity or possibility for such a change. I understand that, even though it is difficult and would obviously require broad consensus, adding more decimal points to bitcoin is possible. This question is not about whether we would ever actually need/want such a change.
I am solely interested in the technical details as to why adding further divisibility requires a hard fork (or evil soft fork) and would not be possible with a "regular" soft fork. I would say I already have a good feeling why, but could not really explain it to someone else.
I have thought about "separating" the new decimal points in some way but that would most likely not work, since old software would not accept if someone wants to spend two 10000(.5) sat inputs to a 20001 sat output. The question of how old software would interpret a transaction with added decimal points in general is the main problem I guess.
However, how can we be sure that there isn't an elegant solution to this? Why would it require a major change of existing bitcoin software?

I don't remember the reference, but I think it was a mandatory soft fork about encasing the new economy in OP_RETURN payloads meant to be ignored by obsoleted nodes but interpreted by the soft fork enabled nodes according to newer more restrictive rules.

I think the answer depends on exactly how you define "forced" soft fork. I find that this concept of a forced soft fork is mostly useful as a demonstration that being a soft fork (in the sense of "miners converge to a single chain as long as a majority enforces the new rules") is unrelated to how invasive a consensus change is to the ecosystem. If we actually want to discuss that impact, it's more informative to discuss it in terms of what old software/features remain functional/available.
Soft forks in general
As I suspect you already realized, it is certainly possible to add precision if you permit any kind of soft fork.
The easiest, and most extreme, way to show this is possible is through the mechanism described in the blog post by Peter Todd you linked above. A "Bitcoin 2.0" is introduced, with possibly entirely new rules, and higher precision, and the consensus rules say that new blocks must be empty apart from a coinbase transaction that contains a hash of the 2.0 block data. Old nodes don't see the new block data at all, and thus anything is possible there. The UTXO set of new nodes is "imported" by copying that of old nodes at activation time, adding the extra precision in the process.
This is a soft fork in the strict sense of the word, but it would render all old Bitcoin software (not just validation nodes, but all miners/wallets/indexers/libraries/infrastructure/...) useless as they won't see the new transactions at all.
As a one-way extension block
A somewhat less invasive - but possibly more complex - approach is possible.
The idea is to still have the "Bitcoin 2.0" block data from the previous section, with transactions that operate on a higher-precision UTXO set, committed to in the coinbase of old blocks. However, old transactions can still remain in the old blocks, visible to old nodes.
In this case, the UTXO set isn't migrated from old to new upon activation, but instead whenever coin owners want to move their funds over. A special address type is defined by the softfork on the old block side, and funds sent to it are burned from the perspective of old nodes, but magically revived on the new side.
Old software remains functional, but can only see and interact with non-migrated funds.
As a two-way extension block
It's possible to go even further, and permit coins to be moved both ways between the old and new world.
Instead of having a burn address on the old side, use a transfer address which looks like an anyone-can-spend output on the old side. To move funds back, a special burn transaction is defined on the new side, and when included in a block, miners are required to construct a transaction on the old side spending some of the outputs sent to the transfer address, bringing them back into circulation there.
This means that funds that are moved over once from the old side to the new side are not stuck there, and it is possible even to move everything back.
However, old software can still not see any transactions or UTXOs in the new world.
By truncating sub-sat amounts
It is possible to go even further on the "transparency-from-old-software" scale
Instead of having two entirely different worlds (different transactions, different UTXOs, different block area, ...), represent transaction outputs as just having the rounded-down-to-integral-sats amount as what is visible to old nodes, but with extra (ignored by old software) data somewhere in the transaction to convey the subsat value. This is about as close as one can get in terms of having old software still see the transactions and UTXO data.
With just that, it does mean that any old-node-visible sat that gets broken up into sub-sat amounts cannot be recombined into old-node-visible sats - the amounts just wouldn't balance out from the perspective of old nodes which violates their consensus rules. If that's not desired, this is probably the simplest possible approach.
This can be mitigated too, but at significant complexity. Any transaction which breaks down sats to sub-sats must have a special (according to old software) anyone-can-spend output with the broken-down sats. Any transaction which recombines sub-sats into sats is then allowed to consume such outputs (but only on the condition, to new software, that the sub-sat amounts balance out) to account for the difference. The complexity is that this effectively implies a shared "wallet" of these anyone-can-spend outputs managed by all new-rules software collectively, with obvious costs and coordination difficulties that implies.
The advantage of an approach like this is that new-software users can send coins to old wallets within the same transaction even as sub-sat outputs.

It requires a hard fork because it would essentially be a redefinition of the base unit.
In Bitcoin, the base unit is the Satoshi. Amounts in transactions are counted with integers in sats. This ensures that there is no way for floating point arithmetic to cause any errors. In order to add more decimal points, this base unit would have to be redefined, and that is a hard fork.
As you pointed out, having the fractional sats in a different value does not work because enough fractional sats become whole sats and then old software will mark such transactions as invalid because the values don't match.

I don't think this is true; an extension-block like softfork can define a new type of transaction, with separate UTXOs that have higher granularity, and special conversion transactions between the two worlds in both ways.

Right, but wouldn't that also split the mempool, blockspace, and UTXO set between these two types of transactions and their outputs?

UTXO set certainly (by definition almost; currently UTXOs cannot store sub-satoshi amounts...). Transactions, mempool, and blockspace would depend on the implementation; e.g. you could imagine a construction where these subsat-transactions are seen as having 0-value inputs and outputs by old nodes, but they do still see the transactions otherwise. I'm not claiming this is a particularly realistic option - just that it's technically possible with a softfork.

But such a soft fork would fall under the category of forced/evil soft forks, correct?

@sutterseba Under the assumption everyone wants it, why would it be evil or forced?

The term "evil" is of course subjective, I am referring to this https://petertodd.org/2016/forced-soft-forks blog post. Since the UTXO set is split, old software must upgrade in order to use the "actual" bitcoin while in a "regular" (lacking for a better term) soft fork this is not the case.

@sutterseba I don't know, I find that this concept of a forced soft fork is mostly useful as a demonstration that "soft fork" on itself has little to do with how invasive soft forks can be. As a definition, I feel it isn't well specified. If you really want to classify express properties of a consensus change besides convergence, it becomes a question of what pieces of software/infrastructure remain functional. See my own answer.


text:
Where is an example of a feature test testing bitcoin-cli?
Looking for an example so I can run some functional tests against bitcoin-cli.

test/functional/interface_bitcoin_cli.py is a test for bitcoin-cli itself.

how did I miss that? Lol ty.


text:
signet network Error: CreateBaseChainParams: Unknown chain sig
What does this mean? I'm trying to work with signet to review some prs and I'm getting a fair amount of errors.

Is it verifying signet blocks to tip or is it stalling/exiting with these errors? Signet blocks are signed by one of the Signet administrators so it must be related to this. Can't yet find the error message in the code though. (I'm assuming this is the default signet and you aren't trying to set up a custom signet that you control)

Yes I'm using default signet. I'm going to try and reproduce it today.


text:
How can I tweak a private key in bitcoin core
I'm wondering what the workflow is for building tapscripts.
I can generate a new address, but how do I tweak it with the tap script?
I've done the bitcoin taproot workshop a couple times now and starting to get a grip of what is going on, but I can't find where I can do the same things in the workshop's provided python libraries with bitcoin core, I'm also wary of generating my own keys, as the workshop says specifically not to do that. "Bad randomness and no side channel protection." to loosely quote.
Could bitcoin core benefit from this added functionality?
Command's like tweak add, Tapleaf construction of csa, hashlock, csa hashlock delay and proper Tapscript construction?
Right now it seems like I would have to implement these features in a separate python library, maybe by calling bindings to secp256k1 lib for safe randomness?
Thanks
Andrew

Are you seeking to do this with real money? I'd recommend not to and do this with signet or testnet coins if it is just for educational purposes. Then you don't need to worry about bad randomness, side channel protection. In the functional tests of Bitcoin Core there is some Taproot functionality in Python. https://github.com/bitcoin/bitcoin/blob/master/test/functional/feature_taproot.py

Yea I would be only doing this on signet. Thanks

I've done the bitcoin taproot workshop a couple times now and starting to get a grip of what is going on, but I can't find where I can do the same things in the workshop's provided python libraries with bitcoin core

I think you need to be clearer on exactly what you want to achieve. There is Taproot code in Python in the Bitcoin Core functional tests if you want to dig a bit deeper for educational purposes in Python. But that isn't set up to be a standalone library to interact with. A Python library that does have Taproot support (and MuSig2 support) is buidl-python. Other Python libraries like python-bitcoinlib, ofek's bit and darosior's python-bip380 don't support Taproot at the time of writing (May 2023) unfortunately.

Could bitcoin core benefit from this added functionality? Command's like tweak add, Tapleaf construction of csa, hashlock, csa hashlock delay and proper Tapscript construction?

Bitcoin Core as a full node implementation already supports verifying any Taproot spend in C++. The Bitcoin Core wallet supports Taproot via the tr() descriptor again written in C++. OP_CHECKSIGADD (CSA) is supported via the multi_a and sortedmulti_a descriptors within the tr() descriptor. Hashlocks isn't at the time of writing (May 2023) supported within the tr() descriptor but there is a PR open (PR 27255) to add Taproot'd Miniscript to the Bitcoin Core wallet which would enable it.

I can generate a new address, but how do I tweak it with the tap script?

A note on the terminology. You tweak the internal public key, you don't tweak the address. How you tweak the internal public key is explained in the Bitcoin Optech Taproot workshop here.

I'm also wary of generating my own keys, as the workshop says specifically not to do that. "Bad randomness and no side channel protection." to loosely quote.

Right now it seems like I would have to implement these features in a separate python library, maybe by calling bindings to secp256k1 lib for safe randomness?

If you are using signet or testnet coins you don't need to worry about bad randomness and potentially losing your coins. But note the randomness for seeds, private keys isn't generated in secp256k1, it is generated in the main Core repo (random.cpp).


text:
Bitcoin blockchain verification on download
I would know what does Bitcoin verify when you download a block.
Are the transactions verified too, or not? If they are, will it be the case for their signatures?
Best wishes.

It depends what kind of software you're talking about.
A full node, by definition, verifies all consensus rules for newly received blocks and their transactions, including:

Proof of work
Difficulty adjustment
Subsidy (do miners not print too much money)
Double spending (do transactions not spend anything already spent)
Transaction balance (do transactions not spend more than they send)
Resource limits (block weight limit, op count limits)
Signature validity
...

Other software that interacts with the Bitcoin network may verify whatever it likes - ranging from all of the above, to some subset, or nothing at all and just trusting what other nodes or servers give it.


text:
Problem calculating routing fees
I'm trying to compute the routing fees for a lightning network payment of 100,000 msats. The route uses the following sequence of channels (CHAN1 is not shown for privacy reasons):
CHAN1 (base fee: 1000,  ppm: 9)
647868x2102x1 (base fee: 0, ppm: 5000)
751905x3128x0 (base fee: 0, ppm: 5000)
751905x3125x0 (base fee: 1000, ppm: 1)
The route that lightning-cli returns is the following:
"route": [
   {
      "id": "0298f6074a454a1f5345cb2a7c6f9fce206cd0bf675d177cdbf0ca7508dd28852f",
      "channel": "CHAN1",
      "direction": 1,
      "msatoshi": 101505,
      "amount_msat": "101505msat",
      "delay": 123,
      "style": "tlv"
   },
   {
      "id": "023d70f2f76d283c6c4e58109ee3a2816eb9d8feb40b23d62469060a2b2867b77f",
      "channel": "647868x2102x1",
      "direction": 1,
      "msatoshi": 101505,
      "amount_msat": "101505msat",
      "delay": 89,
      "style": "tlv"
   },
   {
      "id": "0265c45476ef53365755f7076eef987965f308b04dba1d8f74638e75b3164eafdd",
      "channel": "751905x3128x0",
      "direction": 0,
      "msatoshi": 101000,
      "amount_msat": "101000msat",
      "delay": 49,
      "style": "tlv"
   },
   {
      "id": "026165850492521f4ac8abd9bd8088123446d126f648ca35e60f88177dc149ceb2",
      "channel": "751905x3125x0",
      "direction": 1,
      "msatoshi": 100000,
      "amount_msat": "100000msat",
      "delay": 9,
      "style": "tlv"
   }
]

The second to last channel is routing 101,000 msat, which means that the second to last node (0265c4...) receives 1,000 msats as a routing fee. I don't understand how this number is calculated.
I thought the correct fee is:
math.ceil(base + 100,000 * ppm) = math.ceil(1,000 + 100,000 * (1/1,000,000)) = math.ceil(1,000.1) = 1,001 msats.
What is the correct formula?

I think the confusion is that you assume math.ceil. I believe once everything is expressed in msat everything after the decimal is truncated. so yes while your computation of 1,000.1 is correct the .1 will be truncated. I think you can just think of a casting to int as in int(1,000.1)=1,000 I think msat is just considered to be u_int

Yes, you are right. Thank you, Rene!

Btw, I think the wording in BOLT 7 is confusing: "SHOULD accept HTLCs that pay a fee equal to or greater than: fee_base_msat + ( amount_to_forward * fee_proportional_millionths / 1000000 )". It doesn't specify that this is integer division.

But it specifies all relevant variabls as integers. But I agree it could be more specific. You could open an issue or PR


text:
Failed to start bitcoin daemon using Ubuntu 20.04
Hey I keep getting this error everytime I run sudo systemctl enable --now bitcoind
Job for bitcoind.service failed because the control process exited with error code.
See "systemctl status bitcoind.service" and "journalctl -xe" for details.
When I input sudo journalctl -xe --unit bitcoind I get this:
bitcoind.service - Bitcoin daemon
     Loaded: loaded (/etc/systemd/system/bitcoind.service; enabled; vendor pres>
     Active: failed (Result: exit-code) since Wed 2020-12-30 15:12:05 UTC; 23s >
    Process: 2411 ExecStartPre=/bin/chgrp admin /etc/bitcoin (code=exited, stat>

Here is how my bitcoind.service looks. I changed the user and group from bitcoin to admin.
[Unit]
Description=Bitcoin daemon
After=network.target

[Service]
ExecStart=/usr/bin/bitcoind -daemon \
                            -pid=/run/bitcoind/bitcoind.pid \
                            -conf=/etc/bitcoin/bitcoin.conf \
                            -datadir=/var/lib/bitcoind

# Make sure the config directory is readable by the service user
PermissionsStartOnly=true
ExecStartPre=/bin/chgrp admin /etc/bitcoin

# Process management
####################

Type=forking
PIDFile=/run/bitcoind/bitcoind.pid
Restart=on-failure
TimeoutStopSec=600

# Run as admin:admin
User=admin
Group=admin

# /run/bitcoind
RuntimeDirectory=bitcoind
RuntimeDirectoryMode=0710

# /etc/bitcoin
ConfigurationDirectory=bitcoin
ConfigurationDirectoryMode=0710

# /var/lib/bitcoind
StateDirectory=bitcoind
StateDirectoryMode=0710

# Hardening measures
####################
# Provide a private /tmp and /var/tmp.
PrivateTmp=true

# Deny access to /home, /root and /run/user
ProtectHome=true

# Mount /usr, /boot/ and /etc read-only for the process.
ProtectSystem=full

# Disallow the process and all of its children to gain
# new privileges through execve().
NoNewPrivileges=true

# Use a new /dev namespace only populated with API pseudo devices
# such as /dev/null, /dev/zero and /dev/random.
PrivateDevices=true

# Deny the creation of writable and executable memory mappings.
MemoryDenyWriteExecute=true

[Install]
WantedBy=multi-user.target

Here are the steps I made before:
~$ cd src
~/src$ git clone --depth 1 --branch v0.20.1 https://github.com/bitcoin/bitcoin.git
~/src$ cd bitcoin
~/src/bitcoin$ ./contrib/install_db4.sh `pwd`
~/src/bitcoin$ ./autogen.sh
~/src/bitcoin$ export BDB_PREFIX='/home/ubuntu/src/bitcoin/db4'
~/src/bitcoin$ ./configure BDB_LIBS="-L${BDB_PREFIX}/lib -ldb_cxx-4.8" BDB_CFLAGS="-I${BDB_PREFIX}/include"
~/src/bitcoin$ make
~/src/bitcoin$ sudo make install
~/src/bitcoin$ bitcoind -version
vi bitcoin.conf
server=1
rpcbind=127.0.0.1
whitelist=127.0.0.1
rpcallowip=127.0.0.1/32
zmqpubrawblock=tcp://127.0.0.1:28332
zmqpubrawtx=tcp://127.0.0.1:28333

~$ sudo mkdir -p /etc/bitcoin
~$ sudo cp bitcoin.conf /etc/bitcoin
~$ sudo chmod 644 /etc/bitcoin/bitcoin.conf

Utimately I got

ubuntu systemd[1]: Failed to start Bitcoin daemon.

I think this would better be addressed in the Bitcoin Core issue tracker and I see you've opened an issue. Hence probably best to close this StackExchange question https://github.com/bitcoin/bitcoin/issues/20818

if you installed Bitcoin core using a PPA, it means your script is looking for bitcoind in the wrong place.
To confirm where your bitcoind was installed, use whereis bitcoind
This will likely return /usr/local/bin/bitcoind, now your script is looking for it in /usr/bin/bitcoind so you can either copy it into that directory
sudo cp /usr/local/bin/bitcoind /usr/bin/bitcoind
or change your script to look for it in /usr/local/bin/bitcoind


text:
faster way to get bip44 address from mnemonic?
Long story short my 12 word seed phrase has a passphrase and I forgot the order.
So I did all the permutations of the 12 words to find the valid ones (29939407). So the script below is my way of adding the passphrase and then getting all the legacy addresses.
However, I'm not sure if it's my computer but so far the script I have has only written 3.5 million addresses and it has been running for 12 hours! So I was wondering if anyone could recommend any ways to speed up this script?
from bip_utils import Bip39MnemonicGenerator, Bip39SeedGenerator, Bip39WordsNum, Bip44Changes, Bip84, Bip84Coins, Bip44Changes, Bip44, Bip44Coins

mnemonics_filename = open ('sats.txt')
address_filename = open('sats-address.txt')

mnemonics = open(mnemonics_filename, 'r')
address = open(address_filename, 'w+')

for mnemonic in mnemonics:
    seed_bytes = Bip39SeedGenerator(mnemonic).Generate("satsenchanter")
    bip44_mst_ctx = Bip44.FromSeed(seed_bytes, Bip44Coins.BITCOIN)
    bip44_acc_ctx = bip44_mst_ctx.Purpose().Coin().Account(0)
    bip44_chg_ctx = bip44_acc_ctx.Change(Bip44Changes.CHAIN_EXT)
    for i in range(1):
        bip44_addr_ctx = bip44_chg_ctx.AddressIndex(i)
        b44.write(bip44_addr_ctx.PublicKey().ToAddress()+'\n')

Rewriting it in C (or other compiled language) would probably make a worthwhile difference.

What is the information you have, if you compare the result form 12 word seed phrase permutation plus passphrase. With public key instead of address, I'll be faster.  And the most important thing is that Python can't utilize the full capacity of your CPU because it can't handle the Global Interpreter Lock (GIL). The script made from pure Python can only use 1 core, whereas another language like C or Rust can create a multiprocess script. Using those languages can increase the speed up to 10-30 times faster than Python.

However, waiting for 12 hours is not too long when you consider it in contrast to learning a new language without any foundational knowledge.


text:
What is the correct format for wallet.aes.json?
I have a file (wallet.aes.json) which I think it's a Blockchain wallet backup but when I go to Blockchain import it says Unknown format.
But here is a clue of whats inside the file
{
"address":"008aeexxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"crypto" : {
    "cipher" : "aes-128-ctr",
    "cipherparams" : {
        "iv" : "xxxxxxxxxxxxxxxxxxxxxxxxxx"
    },
    "ciphertext" : "dxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    "kdf" : "scrypt",
    "kdfparams" : {
        "dklen" : 32,
        "n" : 262144,
        "r" : 1,
        "p" : 8,
        "salt" : "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
    },
    "mac" : "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
},
"id" : "xxxxxxx-xxxx-xxxx-xxxx-xxxxxxx",
"version" : 3}

I have the decryption password and so many tools have failed, that's why I am starting to think this is not a blockchain wallet backup or maybe it's an Ethereum keystore or something else.
It's from my old hard drive.
thank you

https://blockchain.info/wallet/wallet-format is not working anymore

What is the correct format for wallet.aes.json?

It seems Blockchain.info / blockchain.com's custodial wallets have used several different formats.
In the github repository for btcrecover you can find some example wallets used for testing. These are in the btcrecover/test/test-wallets/ folder

blockchain-unencrypted-wallet.aes.json
blockchain-v0.0-wallet.aes.json
blockchain-v2.0-wallet.aes.json

blockchain-unencrypted-wallet.aes.json is in text format:
{
    "guid" : "9bb4c672-563e-4806-9012-a3e8f86a0eca",
    "sharedKey" : "728e769e-4c26-4301-a0b8-2db25c564c5a",
    "double_encryption" : true,
    "dpasswordhash" : "ba3b18c73dd213b7c411e91f32e202d68208eca63ffe3e4530e6c19fb1f276a5",
    "options" : {"fee_policy":0,"html5_notifications":false,"logout_time":600000,"tx_display":0,"always_keep_local_backup":false,"transactions_per_page":30,"additional_seeds":[]},
    "keys" : [
{"addr":"1BUoAfr3N6RcH3JArntzoFGfk9hrgU4i5a","priv":"oXndJ0rnJR3XTXAxzIAQvjpYdLD1F5X4aCOVB/lKJzwbdQqcYZ4kH9uHodiYXf+jcDHLItxMh5AaAh76m382/w==","created_time":0,"created_device_name":"javascript_web","created_device_version":"1.0"}
    ]
}

blockchain-v0.0-wallet.aes.json looks like base58 encoded data:
5gMPxas8l5M1oCc7ss7tiPWfctSeunHmCpK18tdZh0dsaWQFLXbFV97qEG/v4Rvb
pY4NMv1UJNgm8e6G9UH6bM3H0LoDRct3itduKOyn7kjiulhLCROaKlg/0Xt442v7
t1VfbySy/vjq6sMGYT9BYkwtWfMclFkisz0G/PciNIozNkeMrUVI+0bEFJSn+JyY
qTJ3ROwSvIs8lWFpE0PGUz0sHz9Eltysz5Xw5aIXxdl/9c3zo7cx4mjoSFZGTc11
PL9F4A8Oi/ybV1ud6BbLvV9K08bRwPNPijvAzmKop02gVeLHLAKF4zopnvV7nvnQ
bolHE6u2HZ9LUVfDSCSLY16ZxGQ7YhBSXp+sLd4zNNiD4u4WfACN5hc+WvdT1kJZ
6veCxUrOt8CHPmNldzWyibCaLh+aOTQjFdo2Qo0UCsZmK7lM2nDnv6TYuP+PhdGp
ymPmKgLWiRwpTqBc88no7VqEAuQcdmUPuMENWkTKj1SbLqkvEl6EWMf8jE+IE1wI
uMtAdVSTsxiAXE0pOdo0icul9KLm1L7WTVtU9W3zaSn9dHl2YVDoMLITpryFrM/9
O4UXdG8UcAkuY0xisGhoLND/AVsJ8NUgpPmdFx5XyEfFSzOuqobeNMQT9wB9iAIs
Z5t0svVXBvZH/XmVhBGvkE+VdLnXxiKEjOT+RpD3lXaf4SRXIWDZGWdF9f4aNKkn
06ULGwB/+4P2NhqfUkEcwsZ9A5NqBRDVgzBY69S85c5LUTod8m+lUp7G6AVQ1zO4
TXSB258825QKNnZtx6X/1IaE/Xv8kV2IVCoCeQxOam2nGIk3EvojEfPHIr2Ysr4l
cPXcV7yTayDvWov5gvYQB5sem68SrgPGncFDLX+UFURnLTod+GwcdB6VsvL3lrp+
3+LtmKIEBIEVijuas5zL3g==

blockchain-v2.0-wallet.aes.json is text/json but almost all the data is in what looks like a base58 string.
{"pbkdf2_iterations":10000,"version":2,"payload":"roYQB5lO6YWIgzRhBSRON2mxemmWHbubCqUrSgOIoa0+2ogaF/OvDN9KunHI9G5HwKOanFBecNW80ldzxbJj8r1KMl3+YKlQR6952wJj5aJwo3SJECwVLuEcx91WndLnnvHnKfhGuh3qPWIN1XziW2KER3Xc+cfbVXHm29OaRN2gW92Tn9ujcYjSc2YShIfTCosRJcz3B6UHndmaTo24QFHjT0gDgS/6Vx26qH7/EPDJeOzu8DqRTpDgbdJXnbAiNKORmQLFxS2qc6oPsE9X8eKtNVze+ItdzRJBoBA6lwbuv5ovejZtLR7yads6xiW3xUauVb1YJQm4pINAbonRNMaTGRcUAKnRdZuEaUy48UdQX1dWzN5aTh6XTtE6mg4T+cxRYy0F+gXeok6/9kX5h6Ubhk4F9IV4EcpSasO8n+qX9kTdCA0B97ebiQipncLf1Wxpobhr9HeC86NR3Y6/AfAlVur/QvsE3OWG/XAssQtK2DDMoS3MKvBtPTsQQw0RFC6NnevIYWp4HEtA4Px774v2FWOpXpdEJypdUwb+AjpTrV1j8DsT2m+c17K4hUjRqwp1sN6FdvwEp3WRQibxFsEOQdLtX/mc+kEENFOV+vbvee209UoZhClLYLpWV6n0jckskpPto9kynn3j5x9O/CbTmunXiLxnRAQvWjK6rugkl5ei8rgT9XZQcZIuApwBUqaxGy5JUd/WMjLKYmPaUg/1SEuMxu8Hr64W6zZXUWoXcDUXwQwm0lu4g18+jFkbXu0jbzUVRFQSc8KNm12GlQGK5cqkTnNGIlyJCoZruW6nZ0Gq/Q8k7XzkutSGWQ+Fg+kdDsIYRKADXQ6D5QDWNSlxwh9Bv9GLZ++bRtmRfMxgvX1tLMLtuwL7T+cj7UtbvCj2LRFTk6GKA+9UaB6qoGiNvjc4cgrhDBKmpKQN03A="}

As you suspected, the data you posted looks like an Ethereum Keystore file. Etherum is off topic here, but see How do I get the raw private key from my Mist keystore file?

The blockchain wallet.aes.json file has many different formats depending on when it was created. You have V1 , V2 , V3 and now V4 all different.
The above file you posted seems not to be a blockchain.com/info format but rather geth/mist or any other ERC20 wallet format as the address field indicated its and ERC20 wallet.
You can find out more about their formats from the github https://github.com/blockchain/

Use https://login.blockchain.com/wallet/import-wallet
Drag the wallet file into the area above. Supports blockchain.info (wallet.aes.json) files.Use your passwords and enjoy


text:
Transfer Bitcoin Core data into a database
I've downloaded Bitcoin data locally via the Bitcoin Core app install and sync. I'm looking to build a system though which users can query their account balance. I'm thinking of building a simple web interface to accept a wallet address and in the backend, I run below commands in the bitcoin-cli console:
createwallet "wallet-1" true
loadwallet wallet-1
importaddress "1CfaunqrVpcXmpLheUVWeSP1KPsKDha1Nb" "" false
getbalance

And return the result back to webUI.
However this approach is not scalable to support say 100s of requests per sec. Therefore I'm thinking of parsing the Bitcoin data out from the .blk files into a query-able database, say MySQL or Mongo or Cassandra etc and run the balance query on the above db. Its a one time job of preprocessing and indexing the data and then I've a query service sitting in front of db to query and return result.
Question

What is the ideal way to transfer Bitcoin data out into a suitable db. Since I'm only interested in supporting balance enquiries, what's the best and easy way to achieve this? Any existing tools I can use?
Which db would be suitable for above purpose. sql or no-sql?

Yes. I plan to have security taken care of. Thats not an issue.


text:
Is this mnemonic cryptography flawed?
Recently I setup a crypto wallet which used a 12-word BIP39 passphrase as security. I noticed something peculiar when I went to setup an additional wallet. The first and last words of the previous wallet I created were exactly the same on the new one. I thought this was odd so I created many more wallets to try and determine if that behavior was just a fluke.
On the next dozen or so wallets I didn't get the same first and last words again, although I did on occasion get one or the other. Next I took the original first and last word then input words randomly that were present in my other created wallets in slots 2-11 to see if the wallet would be valid; they were valid 100% of the time. For the last test I decided to take the first word of my original wallet and repeat it twelve times, which also produced a valid wallet.
Is this typical behavior BIP39 mnemonic passphrase cryptography? If not what would the cause be other than a botched implementation?

BIP39 mnemonics have a checksum. For 12-word mnemonics the checksum is 4 bits which means if you pick 12-words at random the mnemonic should be invalid 15/16 times. This looks like a bad implementation of BIP39

If you are asking about a specific implementation, please mention which one, so that we can avoid it.

@MikeD, Thank you for confirming my suspicions; so for such an issue what would be the worst case scenario of this? I would imagine it would still be difficult to find a wallet that is in use already just because of the amount of possibilities, although I could be totally wrong on that.

@Murch, I'm going to report this to the developer right away since this obviously isn't the correct behavior. Once they have enough time to address or not it I'll report back as to what the token is that uses this dumpster fire to secure it's wallet.


text:
How long is SIGHASH?
I am reading these two posts 1 2, and following step by step.
I almost understand, but I still have some questions.

Before signing, input script should be removed and replaced by pre locking script. In the last, SIGHASH should be appended. in this time, SIGHASH is 4 bytes(0x01000000).
However, after signing, SIGHASH should also be appended. in this time(DER), SIGHASH is 1 byte(0x01), as seen in the raw transaction. So how long SIGHASH is?

in the pizza transaction, the r takes 33 bytes, whit 0x00 in the first. When i use ecdsa in python, i have to delete the first byte 0x00. So when DER process, why not just use 32 bytes.

Hi, could you please use titles that summarize the topic of your question?

So how long SIGHASH is?

The sighash type is a number (it is 1, 2, 3, 129, 130, 131; or since BIP341/BIP342 also 0). It doesn't have a length per se.
When computing the signature hash, the sighash type is serialized as a 4-byte value at the end of the message. In the signature itself, it is serialized as a 1-byte value.
I can't tell you why Bitcoin's creator chose to serialize a value that fits in 1 byte as 4 bytes in the signature hash, but perhaps it was intended for future extensions.

in the pizza transaction, the r takes 33 bytes, whit 0x00 in the first. When i use ecdsa in python, i have to delete the first byte 0x00. So when DER process, why not just use 32 bytes.

Since BIP66 it is a network rule that signatures must use the minimal DER encoding. The pizza transaction predates the introduction of that rule.


text:
How can I protect my funds in the event a quantum computer breaks the ECDLP that Bitcoin relies on?
An interesting question was posted recently, discussing the idea of hashed addresses being (not meaningfully) quantum resistant.
As Andrew's answer on that question points out, paying to a hashed address does not 'quantum-proof' your coins, however there is a mention of using a zero-knowledge proof of a BIP32 seed to recover coins safely, against the threat of a quantum computer capable of breaking the elliptic curve discrete logarithm problem (ECDLP).
If this is possible, then clearly there are advantages to more complex constructions for addresses (the BIP32 key-derivation function being apparently more quantum-proof, in this case).

So my question is: what address constructions are known to provide some level of safety from a quantum computer defeating the ECDLP? How is this safety provided? How can a user create addresses today, to allow themselves the best chance of having their Bitcoin-wealth survive a highly capable quantum computer?
Note that this question is meant to explore what a user can do now to mitigate the risk of the 'worst-case-scenario' of ECDLP being broken suddenly and unexpectedly soon; with time to plan there are obviously less scary ways to switch to a quantum-secure algorithm.
EDIT: for the sake of this question, lets assume that users have unanimously agreed to fork the chain from a certain block that was mined before the quantum-computer attack began, and the fork will lock all coins that are easily stolen by the QC (P2PK outputs, reused addresses, etc). Consider the case in which every bitcoin is stored at an address which will allow safe recovery, suddenly the potential damage to the network is lessened greatly. Even if this is unlikely, I think it is academically interesting.

We're deep in speculative territory here but, theoretically, if the elliptic curve algorithm were broken the entire Bitcoin system would immediately collapse.
Let's consider the case where you've successfully guarded your own funds somehow. How many public keys would still be out there vulnerable to such an attack?
Even though your "base" survived the attack, every other user out there who had UTXO on reused addresses would've been nuked and Bitcoin would have collapsed. You'd be the exceptional survivor. It'd be all over the news, dollar value would've gone to near zero, miners would stop and cut losses and so on.
So, in my view, there is no way to protect yourself individually against the eventual breakage of the elliptic curve discrete logarithm problem. Even if 90% of Bitcoin users had unspent UTXO's in safe unused addresses, the 10% that were hacked would cause immense damage to Bitcoin in a social sense.
Since Bitcoin obeys Metcalfe's law, network value being number of users squared, if enough users dropped out of the ecosystem, due to this kind of development in cryptanalysis, it'd surely deal a major blow to the community.
Not to mention another possibility: such a quantum computer could try brute forcing astronomic quantities of random numbers into private keys all day. Eventually they'd stumble on valuable wallets which could cause the community upheaval of "having broken Bitcoin" and its consequences. So having a known public key wouldn't necessarily be the only attack vector. I'm sure there are other possible attacks using quantum computing, perhaps unknown unknowns that would add immense risk to the system.
Math is the Bitcoin guardian, if it's somehow found to be vulnerable no man would be left standing in my opinion.

I agree with everything you wrote, but this didn't provide an answer to the actual question: how can a user create an address, such that they will be able to safely recover coins in a post-QC world? For the sake of the question, lets imagine that users have all agreed upon a certain pre-QC block that they will fork from, so vulnerable funds have not been stolen / all network value destroyed. Perhaps this situation is unlikely, but this question is still interesting from an academic perspective.

@chytrik It's a nice theoretical question. I don't think there's a way to do anything after QC has broken the system. In a scenario where QC has already broken ECDLP it'd be catastrophic. Of course Core developers won't let it happen, unless something truly revolutionary happens in the near future.

If the onset of the attack was obvious (example, at block X we see every single P2PK UTXO consumed to create a single UTXO), then rolling back to a pre-attack block and forking the network to add safeguards against the QC is possible. I'm really just interested in hearing more about what sort of ideas there are to recover coins safely on this forked chain, such as the BIP32 seed ZKP Andrew mentioned. Evidently there is math that holds up against a QC attack, so how can users employ this math to their advantage, today? I do assume an unexpected QC will cause huge damage, in any case.

@chytrik I understand that you want to look at specific cryptographic technologies to be deployed at that scenario. Fine, but what I mean is this scenario would be unsolvable in practice. It's like debating what you'd do about the swimming pool water after a nuclear attack. Let's assume we're at the point where it's been decided to roll back to a pre-attack block, everything else ceter paribus. This is where my answer above kicks in: all those UTXO's pre-fork are vulnerable. Whatever you derive from the keys, to save those keys, is already broken!

@chytrik So let's say you find a cryptographic solution to convert all those pubkeys to the new system. The attacker also has this data available to him. So post-mortem there is no way to recover this blockchain, it's gone.

I don't think that is necessarily true, as evidenced by Andrew's suggestion of using a ZKP of the BIP 32 seed as a mechanism to safely recover funds. The BIP32 key derivation function is not secured through the ECDLP, so if I understand correctly, a QC that breaks ECDLP would not necessarily be able to work backwards through the BIP32 key derivation to obtain the seed. Thus a user could provide proof that they have first-hand knowledge related to the address in question (a ZKP of the seed), in a way that someone with a QC could not.

@chytrik That's an interesting solution. Curious if QC wouldn't be as efficient against the BIP32 KDF.

Let us continue this discussion in chat.


text:
Buy coins using a australian bank account from Germany
I am German living in Germany but do own an Australian bank account. I would like to use that money to buy bitcoins but somehow am not able to at the big exchange markets.
Whenever I identify myself it assigns me a German profile (which is fine) and only allows money transfer from European countries.
Is there a simple way to do this?

is there's a specific website that you are dealing with?

No. Up to know I tried coinbase, independent reserve, bitstamp and safello but not successful

AFAIK on Binance you just select the recipient's location and then send the money there.

Use one of the Australian exchanges such as coinjar.com or coinspot.com.au. They offer methods such as POLI and BPay (which are specific to Australia) for deposits from an Australian bank account.
The rates are slightly more expensive than coinbase but I doubt there's any easy way around that.

For identification, I need to provide photo ID and proof of address. All Australian exchanges I tried so far (e.g. Independent Reserve), require that my address is in Australia to be able to deposit money.
Is coinspot / coinjar different? Because as far as I can see, I need to provide the same ID information.

From a quick look on the coinjar site they do mention verification from outside Oz https://support.coinjar.com/hc/en-us/articles/115004362186.

Thanks for the link. Unfortunately it states "Because we're able to offer bank transfers to Australian bank accounts exclusively at this stage, it isn't possible to complete ID verification if you're not located within Australia."
So out of luck here...

Your question asks effectively where you can purchase BTC via EFT payment from your Australian bank account.
By the sound of it, the exchanges that you have tried have made the presumptions that will prevent this. I wonder what would happen if you used a VPN to setup your account on an exchange?
There are many VPN's available that let you specify the target exit country (Hola for Chrome for example) - it would be a fairly simple matter to trial.
It may also help if you had an Australian email address, although Gmail and similar are country-generic enough that it shouldn't matter. Using the VPN you could probably signup for a yahoo.com.au email address or similar from your preferred webmail provider. Australian companies like iinet and Bigpond also offer email hosting.
If you were to visit Australia, it would be a simple matter to purchase from bitcoin.com.au who require minimal account setup but, a cash deposit (no EFT).
Also, you could ask your Australian bank to issue you with a Debit Card (Visa/MC logo) which will draw directly on your funds, use that for purchase worldwide.

You can choose to go to the bank and withdraw your money and put it in a German account, which would be very convenient

The simple way:

Transfer money from an Australian bank to a German bank.
Buy bitcoin from a German bank.