text:
Should port forwarding for running bitcoin be done also on the ISP's router?
Follow up question on What is the NAT traversal technique used by bitcoin.
The network architecture described here https://superuser.com/questions/1109859/2-routers-connected-in-series-and-port-forwarding is something that I (probably) have in my home (and probably 99% of other people):
Internet > ISP router (NAT 1) > My home router (NAT 2) > My Node

So my Node is hidden behind 2 NATs. Therefore I should forward ports on both routers to allow incoming connections to my Node.
But people have no access to NAT 1 and can only port-forward NAT 2, and must probably contact ISP to port-forward NAT 1.
How do people run their nodes with incoming connections enabled when this issue is present?
EDIT: In discussion with lenlord it seems that his ISP's router does not perform NAT and that his home router has (dynamically) assigned public Internet address. Is this the standard configuration ISP's provide for homes.
EDIT2: After contacting the ISP and reading some forum posts I found out that I'm indeed behind their NAT (NAT 1 in picture, so called CG-NAT), but that they can remove it for me for free.

is the ISP router installed at your place? You should have access and be able to configure it. Typically, I set the ISP router to operate in bridge mode and just let my router do all the NAT.

I have my home router in my home but there should also be a router that ISP controls that many homes are connected to (and operates as NAT, so ISP does not need to have 100 public addresses for 100 homes in my building).

that's the first time i've encountered such an arrangement. I guess you'll have to ring up your ISP then.

In discussion with @lenlord it seems that his ISP's router does not perform NAT so that his home router has (dynamically) assigned public IP address.
After reading some forum posts and contacting the ISP I found out that some ISPs (and mine is one of them) do indeed hide households behind their NAT (NAT 1 in picture, so called CG-NAT), but that they can remove this NAT for free on demand.
Remove this NAT on the ISP's router is needed to operate bitcoin node with incoming connections.

It is common elsewhere in the world for NAT2 to be only a switch rather than a router in a home setup if it is used at all. It is also just as common there for the customer to have full access to NAT1 which is commonly a wi-fi enabled ADSL/VDSL modem router.

Further, many can use Tor without port forwarding and this will allow also incoming connections when correctly configured.

Whatever is between your computer and your public IP has to be forwarded.


text:
When and how does bitcoin node sync itself?
When does bitcoin node sync itself?
Do different types of sync in bitcoin exist (initial, periodic...)?

Always.
When starting up, Bitcoin Core sends out a request to its peers to inform it about the best chains they know about (using a getheaders message).
After that point, it is automatic: peers will immediately announce new blocks they knew about after they learn about them. This either happens using:

legacy inv mechanism, in which case just block hashes are announced
BIP130 (if negotiated), in which case announcements send full block headers
BIP152 high-bandwidth mode (if negotiated), in which case blocks are immediately transferred in full using compact blocks protocol

I was thinking more about the following: If the node (for some reason) missed a block from main branch or a side branch, he will be out of sync. And he will reject all blocks mined on top of these branches? He must perform some kind of sync. Is this done only when disconected/reconected or periodically?

Your node always works towards the best valid block it knows about. If it learns about a side branch that grows longer than the best block it knew about before, it will download it, and rewind the current block chain to the forking point, and then connect the new best. If the new branch turns out to be invalid, it will revert back to the older tip. All that also happens all the time - we always want to have the best possible block active.

Can we continue in https://chat.stackexchange.com/rooms/67480/bitcoin-chain-progression

What stops me from tampering with a new block (I created), before it is distributed to the rest of the nodes?

@daka That depends on what you mean by tampering. If you modify the block is a legal way before grinding the PoW, it will just be valid. If you make an illegal modification (double spend, theft, inflation, ...), the block will be invalid no matter what. If you modify a block after grinding, you invalidate the PoW, and you'll have to redo it.

"the block will be invalid no matter what" why is that? could you please elaborate on this? thanks

Because all full nodes verify all transaction in a block before accepting. The rule is that the longest valid chain wins; not just the longest one regardless of its contents.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Does full node store all valid forks and "alternative" blocks from blockchain history?
Forks are possible when mining the blockchain and happen "all the time" during mining.
This means the nodes monitoring the blockchain have recorded many forks during blockchain's lifetime, with some of non-main branches long several blocks.
Do nodes store all the branches from entire blockchain history?
EDIT: I read another question, and it is logical that the node should return the transactions from the "alternative" branch they were on to the memory pool (or discard them). But does this happen at the very same time when someone "shows" them longer branch, or they still somehow keep the alternative branches?

Do nodes store all the branches from entire blockchain history?

They will store all branches that they know of so long as the block headers of each branch are fully valid.
The way that Bitcoin Core nodes (and all node software based upon Bitcoin Core) work is that they will ask for an receive a block header before asking for the actual block itself. Using the headers, the node builds a headers chain. So if a block header is valid (i.e. it has a valid proof of work and connects to known previous block headers), it will be stored and added to the chainstate. In this way, a node can know of and keep track of all branches of the blockchain without having the block itself.
Even if a headers chain does not have the most work, it will still be kept around forever and kept track of. Although blocks for that headers chain will not be requested, if any peers have that chain as their actual blockchain and the headers are completely valid, it will still be known to a peer and kept track of.
If a headers chain suddenly begins having invalid block headers, the chain of valid headers will still be kept and tracked; the invalid headers extending that won't be.

But why would I keep track of "obsolete" branches? It is reasonable to track only the branch node is currently on.

Because those "obsolete branches" are completely valid and could potentially become the main chain one day. Without tracking those branches, we would be unable to reorg onto the chain with the most work in the event that we find ourselves on a chain with less work.

But 99.99% of these branches are useless. And we don't need to track them at all. If any of them eventually becomes main chain the node can reconstruct it completely from the point it diverged (and can switch itself).

We don't know what point it diverged from. If you are suggesting that we build the chain by asking for the blocks from a peer, then that is easily DoS attackable.

I disagree. You ask from the peer that provided new block. If he fails to provide we stop. No DoS. Moreover the situation is the same as if we stay behind for couple of blocks from his branch (or even the main branch).

The attack is that a malicious peer can continuously tell you that he is using a different fork causing you to continuously request new blocks which take up more disk space, more CPU and RAM for validation, and more bandwidth. We don't know what the branches are, so we would have to take that peer's word for it and see if his branches are correct. By tracking all branches, we don't need to do that and we know what all the branches are and their current tips.

Let us continue this discussion in chat.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
Passing a wallet to rpcwallet in bitcoind JSON RPC
I want to set a wallet as default wallet in the commands that run with json-rpc calls. the -rpcwallet=blah sets the blah wallet as the wallet which command will run on it; how can I simulate the similar thing with sending a JSON-RPC request?
When I sending the request, for example curl --user myusername --data-binary '{"jsonrpc": "1.0", "id":"curltest", "method": "getwalletinfo", "params": [] }' -H 'content-type: text/plain;' http://127.0.0.1:8332/, but it run on default wallet. how can I pass the wallet as a parameter in my request?

To query a specific wallet, set the URL to be
http://127.0.0.1:8332/wallet/<wallet_name>

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
How to develop a scalable web cryptocurrency robot?
I'm trying to develop a robot that will work in the cloud, basically it will buy and sell crypto-coins according to the strategies defined by the users, but I still can not think of a solution to solve the problem of scalability / slowness.
Let's use a simple example, say we have 100 people using the robot and each robot has created 10 orders, this will give a total of 1000 orders for the robot to analyze and then finalize.
I thought about putting these 1000 orders in a loop and the robot is analyzing one by one, it happens that this process will take a long time, what do you suggest? Is there any way for the robot to parse the 1000 orders in parallel?
Another thing I had thought of is renting a server for each client and each server will get their robot hosted, but I think it would cost a lot of money, what do I do?

I made a mistake, I'll correct it, thank you.

I'm voting to close this question as off-topic because it is essentially a programming question about concurrency.

Based just on the text, the question you're asking is a programming one, so i'll provide a programming answer. This will be heavily from a .NET perspective since no other technology requirements were stated in the question.
Assuming you don't rely on the results of a previously processed one, then you could simply task out each of the assessments. I'm more of a .NET programmer and this wouldn't be hard to accomplish in C# but i'm not sure what language you're writing in.
In addition there are considerations to take into account like how much processing you're doing, the resources available etc. The heavier the processing for each, the more you are attempting to parallel process and the lighter your hardware the slower it's going to be.
The only way to know for sure is picking a language to work with, making sure it can parallel process tasks, then working on setting up your "parse" code. Test with 1.. test with 10.. test with 1000. See what results you are getting with whatever hardware. Then move to testing it on a cloud setup. If it takes 5 seconds on your machine, and 5 hours on the cloud rig, then try beefing up the cloud rig to speed it up.
As you're increasing the resources assigned to handle the processing, you'll get an idea of the cost to run your proposed system. Adjust volume processed, adjust code, etc. You'll have to make decisions about volume, cost of hardware resources, etc to get your desired result.
What it SOUNDS like is that it'll be fine. Doesn't seem overly complex or data intensive.
Note: I'm not making any assessment on intent, viability of the idea, or legality that may or may not surround this issue. The question is about processing, as is the answer.

Sorry for not talking about the language, I really wanted to learn about the concept and your response was very helpful, thank you very much!


text:
descriptor and Custom script
I have a Multisignature address 1-3 with a verification of OP_3, just a simple script to understand custom scripts in Bitcoin
Script definition
#OP_1 <Public Key> <Public Key> <Public Key> 3 OP_CHECKMULTISIGVERIFY OP_3 OP_EQUAL
SCRIPT="5121"$(cat compressed_public_key_1.txt)"21"$(cat compressed_public_key_2.txt)"21"$(cat compressed_public_key_3.txt)"53AF5387"

Now, I want to import my address using descriptor
PK1=`cat compressed_private_key_WIF_1.txt`
PK2=`cat compressed_private_key_WIF_2.txt`
PK3=`cat compressed_private_key_WIF_3.txt`
CHECKSUM=$(bitcoin-cli getdescriptorinfo "sh(multi(1,$PK1,$PK2,$PK3))" | jq -r .checksum)
bitcoin-cli importdescriptors '[{ "desc": "sh(multi(1,'$PK1','$PK2','$PK3'))#'$CHECKSUM'", "timestamp": "now", "internal": true }]'

The problem is I can't have the same address because the script is different.
I will do the signature manually, only for study purposes.
I read https://github.com/bitcoin/bitcoin/blob/master/doc/descriptors.md but I didn't find any help about my problem.
Using Legacy code, I was able to import
bitcoin-cli importaddress $ADDR_P2SH

Where $ADDR_P2SH is the result of my address 1-3.
Is it possible to use a custom script using a descriptor? If I understand, a descriptor is "like a function," so I don't know if a "custom" function is available.
Update
Thanks to Antoine Poinsot, I am posting the solution. Maybe it could be useful to someone.
ADDR_P2SH=`cat address_P2SH.txt`
bitcoin-cli -named createwallet wallet_name="bia" disable_private_keys=true

CHECKSUM=$(bitcoin-cli getdescriptorinfo "addr($(cat address_P2SH.txt))" | jq -r .checksum)
bitcoin-cli -rpcwallet=bia importdescriptors '[{ "desc": "addr('$(cat address_P2SH.txt)')#'$CHECKSUM'", "timestamp": "now", "internal": true }]'

and I get the UTXO with
UTXO=`bitcoin-cli -rpcwallet=bia listunspent 1 101 '["'$ADDR_P2SH'"]'`

Please open a new thread for new questions instead of updating the OP.

Now resolved, I appreciate your time. I've posted the solution to help someone in a similar situation.

OP_1 <Public Key hash> <Public Key hash> <Public Key hash> 3 OP_CHECKMULTISIGVERIFY OP_3 OP_EQUAL

This is not a valid use of CHECKMULTISIGVERIFY. This script would not be spendable.

Is it possible to use a custom script using a descriptor? If I understand, a descriptor is "like a function," so I don't know if a "custom" function is available.

Under tr() and wsh() you can import more complex scripts but those have to be valid Miniscripts. Otherwise, the modern equivalent of the importaddress you were using on legacy wallets would be to use an addr() descriptor. In this case however Bitcoin Core won't know how to spend an output with the corresponding scriptPubKey.

Hello, thanks for your reply. This script is for study purposes only. Anyway, I will build the scriptsig properly in order to spend the transaction. It works with legacy commands. you are talking about "addr(ADDR) (top level only): the script which ADDR expands to." correct? what does mean Top Level only? Maybe I could use even raw(hex)

Top-level means that you can't use it inside another desc. Yes you could use raw() too. What are you trying to achieve? If you want to play with the Script interpreter there are tools out there you might be interested in trying out, such as https://github.com/bitcoin-core/btcdeb

Yes, I love btcdeb, and I use it. My purpose is just for study. I want to import a multisignature 1-3, sign it, and check that OP_3 is equal to OP_3 that I put in my scriptsig. I create the signature manually. I repeat, it's just for fun and understanding what happens under the hood. With the legacy core, it works properly, but I updated to version 26, and I want to migrate it.

I see. Note this won't work because as i put in my answer your use of CMS is invalid. If you pass it public key hashes it'll just fail at VERIFY and never get to execute OP_3 OP_EQUAL. :)

the public key hashes form the redeem script and in scriptSig I put Signature and and OP_3. the script works as you can see in testnet https://tbtc.bitaps.com/d3d661ec439baa845a22078b2ced81d346fe0fcd7c1e12f16980c87438d78961/input/0

The P2SH redeem script used in d3d661ec439baa845a22078b2ced81d346fe0fcd7c1e12f16980c87438d78961's input uses public key with CMS, not public key hashes. Maybe you just made a typo in the OP?

oh! understand now! typo in comment :). Edited, thanks BTW: CMS stands for?

I updated my answer with your hint

CMS = op_CheckMultiSig


text:
Blockchain app says I'm over my limit
I have the Blockchain app on my iPhone which I've been using regularly for years but which in fact I haven't used for the last 4 months. It currently has around $2000 worth of BTC in my private wallet. I have just gone to make a purchase for goods of around $50. I click on Send, select Private Key Wallet, enter the recipient's address but as I type $50 it tells me in red that I am trying to send an amount over my limit. Can anyone tell me what's happening with this?

Solved! I went to my profile and under Security I saw the Backup Phrase wasn't confirmed (even though I had made a note of the phrase when I first opened an account). I clicked on Backup Phrase, took a note of the words (same as I had written down) and then answered the question - what is your 2nd, 3rd and 11th word. After that is put "confirmed" by Backup Phrase. I was then able to send btc.
I'd suggest to the developers that they put a more meaningful message than "over your limit" if this is in fact the issue. And why would it let me send $5 but not $50?! Some rule written somewhere in the software that isn't brought to the user's attention.

To make suggestions to the developers of an app you'd need to post your suggestions somewhere those developers read. This self-help Q&A site isn't an official channel for communicating with employees of Blockchain.com or any of the many other businesses providing Bitcoin related services.


text:
Invoicing with BTC
I am trying to learn if there is a way in BTC to raise an invoice from a friend like in Paypal just by using a public address on the BTC wallet. I have checked the API is all leading exchanges, and none provide a handshake mechanism for "invoicing" another account, so was wondering if there's any other way.
For example:
If Jack owed 100 sats to Bob, is there a way for Bob to raise an invoice to Jack, which Jack can then approve and fulfill the invoice? (given that Bob knows the public address of Jack's wallet)
Is it possible to achieve this? How does it work if Jack's wallet is on any of the exchanges vs the offline wallet?
I'm new to this and trying to learn, so Thank you in advance for your patience and help. I really appreciate it.
Best.
Apologies for not mentioning this before: I'm interested in learning about the lightning-based solution and how can one scale it especially when Bob is a vendor who accepts BTC.

Hi ddesai, you tagged this question with [tag:lightning-network], but don't mention lightning network in your question. Could you clarify whether you're only interested in this functionality between user accounts on a service or whether a lightning-based solution would be of interest?

Bitcoin Blockchain
Many Bitcoin wallets allow you to generate an invoice in the form of a QR code, which is a 2D barcode. This can contain a receiving address, an amount and a message.
How the invoicer gets the QR code to the invoicee is outside the Bitcoin network itself. Originally it seemed to be intended that the invoicee would use their phone to scan the QR-code from the invoicer's device. But I guess many other methods are available.

source -- but warning! do not use QR_code generator websites -- some are malicious.
Note that a wallet typically has many addresses, one per transaction. Some people make a good case that it would have been better to use a name like "invoice reference" instead of "address" for the data item called a Bitcoin address. - but its too late to make that sort of change.

Bitcoin Lightning
Question was tagged LN but invoicing in LN is a separate issue. For example, see

Lightning 101: What is a Lightning Invoice?

This can contain information such as:

When the invoice was created which is useful to see how old an invoice is
The ID of the node that is receiving the payment
When the invoice expires. Unlike bitcoin addresses, Lightning network invoices can expire
A description of this invoice. In our example, the description embedded in the invoice is “suredbits demo invoice”.
A backup bitcoin address. If the payment fails over the Lightning network, you can fallback to a regular bitcoin transaction.
A route to make the payment. Remember, the Lightning Network is different than a blockchain in the sense that you need to find a route to make a payment. The receiver of a payment can suggest a route for you to use to make the payment.

Exchanges / Custodial Wallets
Invoices between two customers of a trusted third-party money-holder is typically conducted outside the Bitcoin or Lighting networks. How this works is completely dependent on the specific proprietary services offered by the trusted third party.

Thank you for your reply. I really appreciate it.


text:
In which format bitcoin protocol send message?
I'm writing a small p2p network and I don't know if I have to transmit message (between nodes) in clear text or in another format ? How is it for bitcoin protocol ?

in clear text or in another format?  How is it for bitcoin protocol?

In binary. With numbers mostly in little-endian byte-order. Over TCP with a conventional target port number.
The details are at https://en.bitcoin.it/wiki/Protocol_documentation
For example:

Message structure

Field Size
Description
Data type
Comments

4
magic
uint32_t
Magic value indicating message origin network, and used to seek to next message when stream state is unknown

12
command
char[12]
ASCII string identifying the packet content, NULL padded (non-NULL padding results in packet rejected)

4
length
uint32_t
Length of payload in number of bytes

4
checksum
uint32_t
First 4 bytes of sha256(sha256(payload))

?
payload
uchar[]
The actual data

Unless message size and volume is important, I would instead use JSON in UTF8 for any protocol I create. You should make your own evaluation.
You might also want to consider the Gossip protocol

Thank you ! What are the benefits of using binary instead of json/text ?

Compactness of data on the wire


text:
Why doesn't Bitcoin use different types of transactions with only one type of address?
I don't understand that, why does bitcoin need to use different types of addresses for making the different types of transactions? Is'nt it possible to generate different transactions having a transaction type configuration while using one address?

why does bitcoin need to use different types of addresses for making the different types of transactions?

I think a fundamental reason is that the recipient (payee) has to be involved in specifying how they want to be paid, they have to be involved in the choice of transaction type. The address is really nothing other than the way in which the payee communicates their needs to the payer. It has to contain a specification that is sufficient for the payer to construct a locking script that the payee has the means to unlock. The address is not used in the Bitcoin network at all.

Is'nt it possible to generate different transactions having a transaction type configuration while using one address?

It is, but the address would be insufficient for a payee to be paid properly when the payer might choose a new transaction type that the payee's software did not yet support. As a payee you'd have to specify not just an address but a list of acceptable payment types too. The address is a concept that was supposed to do the whole of that job.
Again, it is worth remembering that addresses don't exist anywhere in the Bitcoin network protocols and don't exist in the blockchain.

It helps prevent confusion and loss of funds.
Different version bytes are reserved for different uses and result in different address types. For example version bytes 0x00 (mainnet address starts with 1), version bytes 0x6F (testnet address starts with m or n).
Imagine if both mainnet and testnet had the same version bytes. It would be very confusing.
See this answer for more reasons why we have different version bytes: https://bitcoin.stackexchange.com/a/552/26873
As for whether it can be done with a single address type Legacy P2SH (addresses that start with a 3) was probably the closest to a single address type that handled multiple spend cases. It could behave like a P2WPKH via a P2SH script (P2SH-P2WPKH) as well as ability to do multisig, and even other non-standard scripts. But it was a temporary work around until native segwit addresses became the standard which have slight variations as well to distinguish output types.

Because need for changes emerges overtime, and in a decentralized system that relies on consensus you need to insure if the changes are backwards-compatible to those who might not agree with. If they are, you can introduce them without the disagreeing's consent.
But, if they aren't backwards-compatible, you can't force them upgrade. You might choose to introduce the changes, but you'll be forked off the network, which isn't desirable in the end.
As for your question. We have multiple types of addresses, or to be precise, multiple scripts to spend bitcoin. We have P2PKH, P2PK, P2SH, P2WPKH, P2TR etc., because there were needs for that, say, P2PKH alone couldn't give us the solution by its own.

P2PK isn't a way around P2PKH

Didn't say it is. Both P2PK and P2PKH were standard from the beginning. Unless I misunderstood you.


text:
On signet of bitcoin core, why didn't the average time between blocks I mined accord with the time I set?
I'm new here. I met some problem when I use signet on bitcoin core in docker container. I used this command to work out the nbits value to set the average mining time which is 60s.
python3.8 miner --cli="bitcoin-cli" calibrate --grind-cmd="bitcoin-util grind" --seconds=60

But then I use this command to mine blocks, the average mining time is not what I set.
python3.8 miner --cli=”bitcoin-cli -datadir=/data/bitcoin/signet-custom” generate --address=$Address --grind-cmd=”bitcoin-util grind” --nbits=1d30783d --ongoing

I got this:
2022-07-11 18:09:20 INFO Mined block at height 25; next in 4m31s (mine)
2022-07-11 18:13:56 INFO Mined block at height 26; next in 2m40s (mine)
2022-07-11 18:16:37 INFO Mined block at height 27; next in 2m44s (mine)
2022-07-11 18:16:37 INFO Mined block at height 27; next in 2m44s (mine)
2022-07-11 18:19:26 INFO Mined block at height 28; next in 2m40s (mine)
2022-07-11 18:22:08 INFO Mined block at height 29; next in 2m43s (mine)
2022-07-11 18:24:52 INFO Mined block at height 30; next in 2m44s (mine)
2022-07-11 18:27:43 INFO Mined block at height 31; next in 2m38s (mine)
2022-07-11 18:30:22 INFO Mined block at height 32; next in 2m44s (mine)
2022-07-11 18:33:07 INFO Mined block at height 33; next in 2m44s (mine)
2022-07-11 18:35:54 INFO Mined block at height 34; next in 2m42s (mine)
2022-07-11 18:38:41 INFO Mined block at height 35; next in 2m40s (mine)
2022-07-11 18:41:25 INFO Mined block at height 36; next in 2m41s (mine)
2022-07-11 18:44:10 INFO Mined block at height 37; next in 2m41s (mine)
2022-07-11 18:46:53 INFO Mined block at height 38; next in 2m43s (mine)

I'm very confused. Maybe this question is stupid, but I will appreciate it if anybody can help me.
By the way, I wonder if I can use RPC command to control my signet remotely. Thanks a lot!

no question is stupid, but for the vision impaired, when text output is screenshot instead of pasted :)

@Mercedes Sorry for that, I have changed it ^^

I second this, I also noticed the same issue and not found a satisfactory explanation. I played a little with the miner python script and I suspect the calibration is broken in some way, but have not been able to determine exactly in what way yet.

I spent some time figuring out what's going on in the miner script, and here's what I think I have understood so far:

on a signet network difficulty adjustement is similar to what it is on mainnet, ie. there's a difficulty adjustment every 2016 blocks and it can only go at most 4 times each way (either divide difficulty by 4 or multiply by 4
when you use the signet calibration script, you try to figure out what the target (expressed in nbits) should be on your node if you were to generate a block every n seconds instead of the standard mainnet 600s.
When you start your node for the first time, the mining script make sure that you mine at most 4 times faster than the expected 10m (which is 2m30s). Even if you target a much faster interval (say, 1m), it wouldn't help to mine faster than that since the bitcoin client wouldn't divide difficulty by more than 4 at the end of the 2016 blocks of the difficulty adjustment period.
After enough difficulty adjustment, you should start to mine at the expected rate

I find it really confusing and convoluted so far, and the available documentation probably needs an update to explain that in more details. Maybe I'm missing something but if the goal is to tweak the block interval away from the canonical 10min it feels a bit off and unnecessary. Obviously I would be happy to be proven wrong by someone that really understands signet and why it was made this way.
I still need more time to figure it out completely and will update this answer if necessary.

Thank you for your detailed answer, and sorry for noticing your answer so late. I already give up using this signet mode for my experiment. Thank you again!

My turn to make an update: it turned out that we are very likely mistaken about this feature, and that as far as I can tell it is not possible to tweak the block interval on a custom signet.
After a while difficulty will go up and eventually you'll be back at 10 min interval, or if you were too aggressive while calibrating difficulty will eventually become too high and you will mine a lot more slowly (at least that's what happened in my experiment).
tl;dr: you can't alter the rate at which blocks are produced. The calibration feature serves another (unclear) purpose.


text:
What is transaction replay and replay protection?
With everyone and their dog making new blockchains forked from the bitcoin blockchain, there is a lot of talk about replay protection.
What is a transaction replay?
What does replay protection do?

What is a transaction replay

In the context of forks, transaction replay is when a transaction is valid on both sides of the fork. So a transaction can be played (i.e. broadcast) on both chains after the fork and be a valid transaction and confirm. This means that if you intend on sending coins on one fork, you could accidentally end up sending your coins on the other fork as well since someone else could take your transaction and replay it (i.e. broadcast it) on the other chain's network.

What does replay protection do?

Replay protection is something that makes it so that transactions on one chain are invalid on the other chain thus preventing transaction replay. This can be anything from having a blacklisted address or output type (e.g. if an output to address X is in the transaction, consider it invalid) to changing the signature scheme to changing the transaction format entirely. All this does it makes replaying some or all transactions impossible.


text:
Java bitcoinj: equals override for LazyECPoint, how should it be done?
In a drive to improve performance, the bitcoinj library defines the class LazyECPoint whose instances are ECPoint which may not have yet been initialized. Hence because an ECPoint (say point) is essentially the same thing as its lazy counterpart (say lazy), the library feels a call to lazy.equals(point) should return true. However, a call to point.equals(lazy) still returns false, and in fact, if point is viewed as an Object same = point, then lazy.equals(same) is now false. My personal gut feeling is that the library should not be the way it is. I was wondering if a professional developer would care to comment as to what the proper way to deal with this situation is. Is it worth attempting to compare ECPoint's and LazyECPoint's, and if so is it worth adding the complexity to the code so as to remove the sort of quirks I have mentioned? (In fact, ECPoint is not a bitcoinj class, it is part of the crypto library spongycastle so I can't even see how to set things straight, without changing spongycastle's code). I attach a code snippet:
import org.bitcoinj.core.ECKey;
import org.bitcoinj.crypto.LazyECPoint;
import org.spongycastle.math.ec.ECPoint;

public class Test {
  public static void main(String[] args){

    ECKey key = new ECKey();                    // some random (compressed) key
    ECPoint point = key.getPubKeyPoint();       // associated ECPoint
    LazyECPoint lazy = new LazyECPoint(point);  // same, but different type

    System.out.println(lazy.equals(point));     // true
    System.out.println(point.equals(lazy));     // false, equality not symmetric

    Object same = point;
    System.out.println(lazy.equals(same));     // false ...

   }
 }


text:
Installing PHP bitwasp bitcoin library
I am attempting to set up the PHP bitwasp library from the github repository:
https://github.com/Bit-Wasp/bitcoin-php
The installing instructions suggest I simply execute:
$ composer require bitwasp/bitcoin
I do have composer installed but I am getting the following error message:
Problem 1
    - Installation request for bitwasp/bitcoin dev-master -> satisfiable by bitwasp/bitcoin[dev-master].
    - Can only install one of: bitwasp/bitcoin[v0.0.29.1, dev-master].
    - Installation request for bitwasp/bitcoin v0.0.29.1 -> satisfiable by bitwasp/bitcoin[v0.0.29.1].
I do not practice PHP full time so I lack the experience of installing packages in that language and using composer. I would be grateful if someone could point me in the right direction.
EDIT
The suggestion has been made that the package is already installed. However, the following code sample which I got from another thread:
<?php
// failing to install bitwasp
use BitWasp\Bitcoin\Bitcoin;
use BitWasp\Bitcoin\Address;
use BitWasp\Bitcoin\Key\PrivateKeyFactory;

$network = Bitcoin::getNetwork();

$privateKey = PrivateKeyFactory::create(true);
$publicKey = $privateKey->getPublicKey();
$address = $publicKey->getAddress();

?>

is giving me PHP Fatal error:  Class 'BitWasp\Bitcoin\Bitcoin' not found.

Did you include the vendor/autoload.php file? It's missing from your snippet, so as suggested below, it might already be on your system.

You have made me very happy !!! First of all I had to type composer install instead of composer require bitswap/bitcoin and this triggered a proper installation, and in particular this created a vendor directory with an autoload.php file in it. Then I created a symbolic link from my working directory to this vendor directory. Then I added the line require 'vendor/autoload.php'; in my snippet as you suggested. And voila the thing ran :) Thank you very much !!. If you want to explain the answer for the record (better than I can do, I am no PHP specialist) I will gladly upvote it.  .

Oh, please be careful :P someone published bitswap/bitcoin on composer and I'm not sure if they're typo-squatting! If you can, @Sven could you update your comment if this wasn't what you meant?

require 'vendor/autoload.php'; For those having similar issues.

You have it installed already. That's why it's saying you have the requirements met and don't need to download it again.

Thank you very much ! I am guessing something else is wrong too. I have updated my question to explain.

I think I know why. Did you use "bitcoin" instead of "bitcoin-php"?

I typed composer require bitwasp/bitcoin to install and the code is exactly as I have stated it. Using the indicated repository. I am on Debian 8.

Oh, im not familiar with Debian. What php version are you using?

PHP 5.6.24. Zend Engine v2.6.0 with Zend OPcache v7.0.6-dev. This is pure PHP, run as a script (no browser involed), if this matters.


text:
bitcoin-qt (master branch) shutdown delay on debian stretch
Following a bitcoin core issue on debian stretch which was resolved on the master branch, I am now running a pre-release test build. Each time I exit bitcoin-qt, I have to wait about 5 minutes until final shutdown occurs. My debug file is showing:
2016-12-20 19:58:55 AcceptToMemoryPool: peer=3: accepted 2b8c3...5c14 (poolsz ...
2016-12-20 19:58:56 AcceptToMemoryPool: peer=3: accepted 4a3d1...7e0e (poolsz ...
2016-12-20 19:58:56 tor: Thread interrupt
2016-12-20 19:58:56 scheduler thread interrupt
2016-12-20 19:58:56 torcontrol thread exit
2016-12-20 19:58:56 opencon thread interrupt
2016-12-20 19:58:56 addcon thread interrupt
2016-12-20 19:58:56 msghand thread interrupt
2016-12-20 19:58:56 net thread interrupt

This is where it hangs for a few minutes, then:
2016-12-20 20:05:58 Imported mempool transactions from disk: 10932 successes, 476 failed, 0 expired
2016-12-20 20:05:58 Shutdown: In progress...
2016-12-20 20:05:58 Stop
2016-12-20 20:05:59 Erased 1 orphan tx from peer 5
2016-12-20 20:06:01 Dumped mempool: 0.063746s to copy, 1.55051s to dump
2016-12-20 20:06:01 Shutdown: done

I have not experienced this before ever on any system. I am not seeing a similar issue when shutting down bitcoind (EDIT actually it does happen with bitcoind at times). I am wondering if anyone is aware of any pending change (on master branch for future release) which would explain such behavior (I worry this may be related to debian stretch, but my hardware is new, very cheap and hopelessly slow on known benchmarks, which may also be the cause of this).

Please report this as a bug.


text:
Why are transactions grouped into blocks?(puzzle creation)
Why are transactions grouped into blocks (PART 2)?

Let's say we have just put a block on a chain. At that moment does a protocol create a new unique puzzle to be solved? Is the new puzzle known before that moment? Can different puzzles coexist at the same time (so that users solve different puzzles in parallel)?

EDIT: Puzzle in this sense means a combination of target (that changes every 2016 blocks), a hashing function, and other protocol imposed requirements miners use to guess the solution.

As a minor rant, my personal opinion is that "informal" Bitcoin texts, although well meant, are responsible for a vast amount of confusion and misunderstanding.  Metaphors like "puzzle" invariably give a mistaken impression about something crucial.  People whose goal is to get a precise understanding should go straight to the formal technical documentation.

Well, for me word "puzzle" was very clear :).

The "puzzle" metaphor may not be helpful.  The goal is always the same: create a block whose header has a sufficiently small hash value, less than the current target.
So every miner can create a block any way she wants, as long as it meets the general requirements of the protocol (all transactions are validly signed, block reward is no larger than the current standard value, etc), and then start varying the nonce value in hopes of finding one which makes the hash sufficiently small.  There is no single centralized authority or protocol that hands out blocks for people to work on.
In practice, this means that no two miners are ever working on exactly the same block.  Two different solo miners will have different blocks, because each will have a coinbase transaction that pays the reward to his own address.  Two different members of the same pool will be assigned different variants of the same block, in order to avoid duplication of work (perhaps each will put his own identifier somewhere in the block).
The target is, in some sense, uniquely determined by the protocol.  The Bitcoin protocol specifies how the target is to be computed, based on the timestamps of prior blocks; this is what ensures that it adjusts with the total mining power of the network to keep the block time approximately constant at 10 minutes.  It should be the case that all miners on the network are using the same target value to compare with their blocks.
So in that sense, everyone is trying to solve the same "puzzle" (aiming for the same target), but everyone is trying a different approach to do so (a different candidate block).
The target only changes every 2016 blocks (approximately every two weeks).  So most of the time, you can know what the target will be after the next block: same as the current target.  If the next block would be a multiple of 2016, then you will need to wait until it is mined, because its timestamp will be involved in the computation of the next target.

I understand that protocol does not give block to mine, but I thought it gives the new puzzle when some block is put on the chain. That users create blocks and try to solve this puzzle. Well, can't it be said that the puzzle (to guess a valid hash) is given by the protocol?

Are you talking about the target?  The "puzzle" is to create a block whose header has a hash value which is less than the target.  It's true that the protocol defines the algorithm by which the target shall be calculated, based on the times at which prior blocks were mined, and so in that sense the protocol "gives" the puzzle, but the way you're saying it is rather confusing.

I explained myself in the edit of comment, and we are at the same page I think.

@croraf: I kind of wish you would stop using the word "puzzle".  It's not a standard technical term in the Bitcoin community, so people are not going to clearly understand what you mean by it, and confusion will result.  If you're reading some source that uses this term, I would suggest you look for something that uses more standard and precise language, since obviously you are looking for precise explanations.

Give me better term and I'll use it :). So my original question with "puzzle" changed with "target", can you answer that?

@croraf There really is no puzzle or anything analogous. There is not anything which the protocol gives to miners to do; miners make the blocks themselves to mine it. The protocol gives miners a goal: to create a block which has a hash less than a give target. The protocol gives the target.

I consider this a "puzzle" to be solved: "to create a block which has a hash less than a give target"

@croraf: I am not quite sure what your question would mean if you replace "puzzle" with "target", but I edited my answer to try to address it.

One comment on your updated answer. I don't think the target is the only thing that defines the "puzzle" but also a hashing function. (Maybe I'm saying something stupid :D). And that after each mined block a new hashing function is given miners use to achive the target?

@croraf: True, in some sense the hash function is part of the puzzle, but it never changes.  The hash function is always SHA256D.  Also part of the "puzzle", in that sense, are the protocol's requirements for what makes a valid block, including the fact that it shall contain the hash of the previous block.

This is hashing family of functions I would say. But the actual functions change each block? Could this be correct? If not, solution of this block gives me the solution of every next block. Maybe I'm mixing terms - but something like that.

I don't think that's a correct way to put it.  SHA256D is a single particular function which takes an input of arbitrary length and outputs a 256-bit hash.  We always use this function.  But the protocol does impose particular requirements for how the input ought to look, and the values do change from one block to the next (most obviously, the hash of the previous block).  As such you cannot use the same block twice in the chain.

Cool, thx! For more details I will check in documentation if needed.

I just posted third part of the series of questions :).

Lets say we have just put a block on a chain and there are no blocks at the moment that users mine. At that moment does a protocol create a new unique puzzle to be solved? Is the new puzzle known before that moment? Can different puzzles coexist at the same time (so that users solve different puzzles)?

As I said in my previous answer, this is not how mining works. There is no central authority or protocol which hands out blocks to miners for them to work on. Rather each miner creates their own blocks and works on them. Each miner works on a unique block that is created independently of other miners.

Yes I understand that. I just thought that a new puzzle is created by protocol when a block is put on the chain. And then the miners try to solve that puzzle by creating their own blocks (with a value that can possibly solve the hash puzzle).


text:
What is the strategy mining pool servers use (regarding difficulty)?
If I understand correctly mining pool servers give miners target thresholds of lower difficulty so each miner can mine a number of 'block solutions', called shares.
For example if threshold is  0000 1000, mining pool can give each user the threshold 0100 0000, with 1000 times lower difficulty. Then every cca 1000 mined shares will give one actual mined block.
The actual target threshold in the block header stays as original as having lower difficulty in block header would invalidate original block hash and lower target threshold is communicated separately?
Is each miner given the same difficulty?
Is client's hashing power taken into account when assigning work?

Possible duplicate of Work distribution in a mining pool

@MCCCS the proposed duplicate doesn't mention difficulty, so this question should remain open.

This is true, although my question is not of high value, and the proposed duplicate helped me not to post another duplicate I wanted to post. I edited the question now to be a bit more relevant.

It depends on the pool. Some pool use a fixed difficulty. Some use several and allow users to choose. Some automatically set the difficulty, with the aim that each miner generates a share every few seconds.
If the difficulty is too low, network traffic is too high (and some miners, such as the antminers, cannot generate more than 40-50 shares/s, regardless of the difficulty).
If the difficulty is too high, then miners generates shares too infrequently, and it may lead to a lost of efficiency, because the job changes every 10 minutes (each time a new block is mined).


text:
How is mining difficulty defined (formula)?
The result of sha256(s) that miners attempt is a 256 bits long number (64-digits hexadecimal number).
I understand that this 256bits number should be lower than the target threshold, the target threshold being itself a 256 bits long number with a number of leading zeros.
Target threshold is encoded with 4 bytes in the block's headed, as 0xEEMMMMMM in hex notation. EE being the 1byte exponent and MMMMMM being the 3bytes mantissa.
How exactly are these exponent and mantissa translated to the actual 256-bits long target threshold?

The nBits field is basically scientific notation in base-256 (256 is 2^8). As an example, we take the one found on the Bitcoin.org Developer Reference: 0x181bc330 (Big-Endian order). This is split into two parts, the 0x18 exponent (24 in decimal), and the 0x1bc330 mantissa. The mantissa is 3 bytes long, so subtract 3 from the exponent and then raise 256 to that power, and multiply by the mantissa just as in scientific notation:

0x1bc330 × 256 ^ (0x18 - 3)

Giving the target, the mantissa followed by 21 0x00 bytes, or 42 zeroes in hexadecimal.
See this answer for another example: https://bitcoin.stackexchange.com/a/61967/51948
EDIT: You can also think of it as allocating the number of bytes given in the exponent, in this case 0x18=24 bytes, and than filling the first 3 bytes with the mantissa.

I like the explaination in the edit.


text:
Naming in segwit script
I have some doubts about naming in witness script/transaction.
P2SH-P2WSH (multisig)
WITNESS_SCRIPT = OP_3 <public key1> <public key2> <public key3> <OP_2>
(is it witness script or redeem script? For now I will call Witness script)
SCRIPTHASH= SHA256 (WITNESS_SCRIPT)
is it Witness script Hash or redeem script Hash?
REDEEM_SCRIPT=0020+SCRIPTHASH
is it redeem script? Maybe it's the redeem script because it is in Scriptsig. (Witness version + Witness program)
In scriptPubKey
hash160 of (0020+SCRIPTHASH).
What is its name? Hash160 of ?
And We have OP_HASH160 <Redeem script hash?> OP_EQUAL
P2SH-P2WPKH
KEYHASH = hash160(<public key>)
REDEEM_SCRIPT=0014+KEYHASH
is it redeem script or witness script? For now I Will call redeem script.
In scriptPubKey
hash160 of (0020+SCRIPTHASH).
What is its name? Hash160 of ?
And We have OP_HASH160 <Redeem script hash?> OP_EQUAL
P2WPKH NATIVE
WITNESS SCRIPT = hash160(<public key>)
In scriptpubkey
0014+WITNESS_SCRIPT
OP_0 <20 bytes witness script hash>
P2WSH NATIVE
WITNESS SCRIPT = <OP_2> <pubkey1><pubkey2><pubkey3><OP_3> <OP_CHECKMULTISIG>
WITNESS_SCRIPT_HASH = HASH160 (WITNESS SCRIPT)
In scriptpubkey
0020+WITNESS_SCRIPT_HASH
OP_0 <32 bytes witness script hash>
P2WPKH NATIVE and P2WSH NATIVE should be ok

P2SH-P2WSH:

Witness script: OP_3 <public key1> <public key2> <public key3> OP_2 OP_CHECKMULTSIG
Witness program: SHA256(witness script)
redeemScript: OP_0 <witness program>
Script hash: Hash160(redeemScript)
scriptPubKey: OP_HASH160 <script hash> OP_EQUAL

P2SH-P2WPKH (I believe that's what you meant instead of "P2WSH-P2PKH"):

Witness program: Hash160(public key)
redeemScript: OP_0 <witness program>
Script hash: Hash160(redeemScript)
scriptPubKey: OP_HASH160 <script hash> OP_EQUAL

P2WSH-P2PKH (native P2WSH, but with a P2PKH script inside):

Witness script: OP_DUP OP_HASH160 <Hash160(pubkey)> OP_EQUALVERIFY OP_CHECKSIG
Witness program: SHA256(witness script)
scriptPubKey: OP_0 <witness program>

Note that you should use P2WPKH instead of this.
Native P2WPKH

Witness program: Hash160(public key)
scriptPubKey: OP_0 <witness program>

Native P2WSH

Witness script: OP_3 <public key1> <public key2> <public key3> OP_2 OP_CHECKMULTISIG
Witness program: SHA256(witness script)
scriptPubKey: OP_0 <witness program>

Thanks again for your time, it's an amazing answer for me. I don't understand this script P2WSH-P2PKH the third one. I create a bash script to create P2WSH-P2PKH native (bc) where the Witness script is OP_DUP OP_HASH160 <Hash160(pubkey)> OP_EQUALVERIFY OP_CHECKSIG. But I have the last scenario (Native P2WSH), than I don't have redeem script and script hash.thanks for your advice (Note that you should use P2WPKH instead of this.) but my purpose now is to study, I don't use it :).

I have another question (sorry). You said "P2SH-P2WSH" redeemScript: OP_0 <witness program>. When I use signrawtransactionwithkey I need to put redeemScript in witnessScript value. Then why use the name redeemScript when I need to use in witnessScript value during signrawtransactionwithkey?

Update last question. If I change WitnessScript with redeemScript in signrawtransactionwithkey works as well. ( I mean the Object key )

@monkeyUser signrawtransaction* secretly doesn't really care about what you name things; it just needs to know all the scripts involved in order to sign.

I don't understand your first question (about P2WSH-P2PKH).

When I create P2WSH-P2SH (Native and with P2PKH script) I have the same items of Native P2WSH (multisig).

What do you mean by "items"? There would not be any redeemScript in the native witness cases, only in P2SH-wrapped cases.

you said "P2WSH-P2PKH (native P2WSH, but with a P2PKH script inside)". It's native, for that reason I don't understand Why I should have redeemScript Script hash. Thanks

Oh, my mistake! Fixed.

thanks for your time, really.


text:
Limitations in P2WSH custom script
I did a stupid custom validation script in P2SH where after check signature, check if OP_3 is equal of OP_3. Now I want to move the same logic in P2WSH multisignatue 1-3.
it's my script
$ bitcoin-cli decodescript 512103dee28d1db5d8b92181ac5baa04c28c01d2b14d19a3c6adaf7bd3430d0c6c8146210343a6f95f841b48ba2b97822a546976aea951e5a1f1c0a10078fdbbded165dbe32103f2f0a25413bb97af3b2ef21694fab820146e2e1bed41b6528a9ce6dd218d9bd353AE755387
    {
      "asm": "1 03dee28d1db5d8b92181ac5baa04c28c01d2b14d19a3c6adaf7bd3430d0c6c8146 0343a6f95f841b48ba2b97822a546976aea951e5a1f1c0a10078fdbbded165dbe3 03f2f0a25413bb97af3b2ef21694fab820146e2e1bed41b6528a9ce6dd218d9bd3 3 OP_CHECKMULTISIG OP_DROP 3 OP_EQUAL",
      "type": "nonstandard",
      "p2sh": "2N6SWbfkKVhPhYXx9idgDG5AVTPSAMNG5tY",
      "segwit": {
        "asm": "0 e976e1bb7c72f7f82132c706b3e8f2ee5673b9fa7172035aab9be62e8877d37c",
        "hex": "0020e976e1bb7c72f7f82132c706b3e8f2ee5673b9fa7172035aab9be62e8877d37c",
        "reqSigs": 1,
        "type": "witness_v0_scripthash",
        "addresses": [
          "bcrt1qa9mwrwmuwtmlsgfjcurt868jaet88w06w9eqxk4tn0nzazrh6d7qm9wxcj"
        ],
        "p2sh-segwit": "2NCLN7WkLL7dt56XyMVWH3QxDv2ytJGEw6d"
      }
    }

then create the transaction, manually sign it and add 3 in my Witness field, Then I have
"5300"<signatures><witness script>

But I get an error during send transaction TX decode failed
The I change 0x5300 with 0x010300 and it works.
The Question is, are there some limitation on custom script in P2WSH? I didn't find in BIP.
In P2SH I able to add 0x53 in scriptSig and send transaction without problems

Perhaps unrelated to your question, but how do you expect a script like this to work? The OP_FROMALTSTACK operation will fail the script immediately, because the alt stack is empty at that point.

@PieterWuille my mistake, I pasted even the length. Updated :)

There is no restriction due to segwit, but witnesses are serialized differently than scriptSigs.
A scriptSig is a script on itself, which gets executed, and its resulting stack is then fed to the scriptPubKey (or redeemScript in case of P2SH). This is not the case for the witness: it is an encoding of the stack directly, without script involved.
That means that scriptSig can use opcodes like OP_3 (which pushes the byte array 0x03 onto the stack). In the witness, you'd need to serialize that byte array directly as 0x0103. The 0x01 there is not an opcode; it is just the length of the stack element.

thanks! I presume that for the same reason witness field doesn't have length in transaction data, but just the #stack item to push in.

Correct. Scripts are serialized as a byte array (so they're prefixed with their total length), and then executed by interpreting opcodes. Witnesses are serialized as an array of byte arrays (so prefixed overall with the number of stack elements, and then each stack element is prefixed with its length).


text:
Confusion about Addresses
I have a little bit confusion about addresses segwit and not segwit.
I can see P2WPKH and P2WSH from this link https://en.bitcoin.it/wiki/BIP_0173, and they are segwit and start with bc (mainnet) or tb (testnet).
But I can see even P2SH, start with 3 nested P2WPKH or P2WSH, and that address start with 3.
Then When we speak about P2WPKH and P2WSH we mean Native and start with bc, or We can have even P2WPKH start with 1 and P2WSH start with 3?
when I create
bitcoin-cli getnewaddress "segwit" "p2sh-segwit"

which kind of script I have? P2PK?

For more information about address types, see my github writeup on address derivation.

6 types of Bitcoin addresses (in parentheses are the data they keep):
Legacy

P2PKH 1... (a public key's hash)
P2SH  3... (a script's hash) (defined in BIP16)

SegWit (P2WPKH/P2WSH) nested in P2SH (defined in BIP141)

P2SH-P2WPKH 3... (a public key's hash)
P2SH-P2WSH 3... (a script's hash)

(these start with three because these addresses are meant to be compatible with existing wallets, who would see those addresses as P2SH wallets)
Native SegWit (P2WPKH/P2WSH) (defined in BIP141 + BIP173)

P2WPKH bc1... (a public key's hash)
P2WSH bc1... (a script's hash) (but longer than P2WPKH)

which kind of script I have? P2PK?

getnewaddress can't return P2PK because you can either pay directly to a script (which can be P2PK) or a script enclosed in P2SH (or its derivatives P2SH-P2WSH and P2WSH). But that's not very useful.
The documentation says:

Arguments:

"label"          (string, optional) The label name for the address to be linked to. If not provided, the default label "" is used. It can also be set to the empty string "" to represent the default label. The label does not need to exist, it will be created if there is no label by the given name.

"address_type"   (string, optional) The address type to use. Options are "legacy", "p2sh-segwit", and "bech32". Default is set by -addresstype.

The legacy type means P2PKH, p2sh-segwit means P2SH-P2WPKH and bech32 means P2WPKH.

In P2SH-P2WPKH that I can get with getnewaddress P2WPKH has the bech32 logic inside (https://en.bitcoin.it/wiki/BIP_0173)? is it native bech32 but wrapped into P2SH?

The goal of SegWit addresses is to move signatures to a register named "witness". Because both P2WPKH and P2SH-P2WPKH do that, both are SegWit addresses. But a native SegWit address is more space efficient than a P2SH-nested address; it saves about twenty bytes (examples). And Bech32 is the serialization method to represent native SegWit addresses. So both are SegWit addresses but Bech32 is more SegWit as it doesn't aim to be backwards-compatible (so old wallets can't send to it but it can send anywhere)

So slightly different mechanics.

thanks again, here https://github.com/bitcoin/bips/blob/master/bip-0141.mediawiki#examples I can See P2WPKH not inside P2SH. In that case what kind of address I have? With 1 ?

If it doesn't mention P2SH and it's SegWit then it's a native address, represented by Bech32. In mainnet all Bech32 addresses start with bc1 (unless they change it in the future).

Then P2SH-P2WPKH, P2SH wrap a native P2WPKH, right? thanks again for your time man

No they have different spending ways (different scriptSig and scriptPubKey). Another way to think of it as is that a P2WPKH is not a script (there is no CHECKSIG, etc opcodes recorded in the blockchain) therefore it can't be nested in P2SH therefore P2SH-P2WPKH is a completely different format. You're welcome, feel free to ask if you have more questions.

how can I create P2WPKH not native?

The command you wrote in the question does that

Than I don't understand the different between P2WPKH and P2WPKH nested in BIP16 P2SH and How create these addresses. If with this command bitcoin-cli getnewaddress "segwit" "p2sh-segwit"  I create P2WPKH How can I create P2WPKH nested in BIP16 P2SH? both address are inside P2SH, then both are nested

bitcoin-cli getnewaddress "segwit" "p2sh-segwit" is P2SH-P2WPKH. bitcoin-cli getnewaddress "segwit" "bech32" is native P2WPKH.


text:
BlockCypher API signature bitcoin transaction Error
I'm trying to make a bitcoin transaction using the BlockCypher API and I'm getting an error when I try to sign the transaction.
This function creates an unsigned transaction:
def buildTransaction(fromAddress, toAddress, value = -1, feePreference = "low"):

    # 'feePreference' Can be high, medium or low.
    # 'value' in this transaction output, in satoshis.
    # 'value' will be set to -1 to sweep all value from input address

    data = {
        'inputs': [{
            "addresses": [fromAddress],
        }],
        'outputs': [{
            "addresses": [toAddress],
            "value": value
        }],
        "preference": feePreference
    }

    data = json.dumps(data)
    print(data)

    return blockcypherAPI.request('/txs/new', 'POST', data).json()

The function return object:
    {
        "tx": {
            "addresses": [
                "CEztKBAYNoUEEaPYbkyFeXC5v8Jz9RoZH9",
                "C1rGdt7QEPGiwPMFhNKNhHmyoWpa5X92pn"
            ],
            "block_height": -1,
            "confidence": 0,
            "confirmations": 0,
            "double_spend": False,
            "fees": 142700,
            "hash": "afceb017a0d280a746ad5ed5c18228e302ebc2aa71ec768454b9a768f73c24c1",
            "inputs": [
                {
                    "addresses": [
                        "CEztKBAYNoUEEaPYbkyFeXC5v8Jz9RoZH9"
                    ],
                    "age": 163376,
                    "output_index": 0,
                    "output_value": 23455,
                    "prev_hash": "457a538479fb50f3e060702594ef92962c2570d592b8e7c089d7b13fc9b5b3dc",
                    "script": "",
                    "script_type": "",
                    "sequence": 4294967295
                },
                ...,
            ],
            "lock_time": 0,
            "outputs": [
                {
                    "addresses": [
                        "C1rGdt7QEPGiwPMFhNKNhHmyoWpa5X92pn"
                    ],
                    "script": "76a9145fb1af31edd2aa5a2bbaa24f6043d6ec31f7e63288ac",
                    "script_type": "pay-to-pubkey-hash",
                    "value": 1000000
                },
                ...,
            ],
            "preference": "high",
            "received": "2015-10-21T19:11:35.774722671Z",
            "relayed_by": "207.38.134.25, 127.0.0.1",
            "size": 775,
            "vsize": 775,
            "total": 3773210,
            "ver": 1,
            "vin_sz": 17,
            "vout_sz": 2
        },
        "tosign": ['798097d46807966f214079ae99a4989b6b320befc0212f88779437962b2f2585']
    }

The problem starts in this function:
I need to use my private key to sign the data provided in the tosign array from what the previous function returned.
I am currently using the BlockCypher library to sign the
transaction
def signature(unsigned_tx, privKey):

    privkey_list = [privKey]
    pubkey_list = [unsigned_tx['tx']['inputs'][0]['addresses'][0]]
    #print(pubkey_list)
    return make_tx_signatures(txs_to_sign=unsigned_tx['tosign'], privkey_list=privkey_list, pubkey_list=pubkey_list)

The function returns the following error:
Traceback (most recent call last):
  File "/Users/user/PycharmProjects/blockcypherAPI/transaction.py", line 44, in <module>
    print(transaction(fromAddress= "CEztKBAYNoUEEaPYbkyFeXC5v8Jz9RoZH9", fromPriv= "{PK}", toAddress= "C1rGdt7QEPGiwPMFhNKNhHmyoWpa5X92pn", value= -1))
  File "/Users/user/PycharmProjects/blockcypherAPI/transaction.py", line 41, in transaction
    sign = signature(unsignedTX, fromPriv)
  File "/Users/user/PycharmProjects/blockcypherAPI/transaction.py", line 35, in signature
    return make_tx_signatures(txs_to_sign=unsigned_tx['tosign'], privkey_list=privkey_list, pubkey_list=pubkey_list)
  File "/Users/user/PycharmProjects/blockcypherAPI/venv/lib/python3.10/site-packages/blockcypher/api.py", line 1596, in make_tx_signatures
    assert ecdsa_raw_verify(tx_to_sign, der_decode_sig(sig), pubkey_list[cnt]), err_msg
  File "/Users/user/PycharmProjects/blockcypherAPI/venv/lib/python3.10/site-packages/bitcoin/main.py", line 507, in ecdsa_raw_verify
    x, y = fast_add(fast_multiply(G, u1), fast_multiply(decode_pubkey(pub), u2))
  File "/Users/user/PycharmProjects/blockcypherAPI/venv/lib/python3.10/site-packages/bitcoin/main.py", line 193, in decode_pubkey
    if not formt: formt = get_pubkey_format(pub)
  File "/Users/user/PycharmProjects/blockcypherAPI/venv/lib/python3.10/site-packages/bitcoin/main.py", line 174, in get_pubkey_format
    else: raise Exception("Pubkey not in recognized format")
Exception: Pubkey not in recognized format

In addition, I would like to understand how to perform the signature independently without the blockCypher library
Thanks for the help!


text:
Python unable to install Bitcoinlib
I try install bitcoinlib python package and I get an error:
Collecting bitcoinlib   Using cached bitcoinlib-0.6.5-py3-none-any.whl (315 kB) Requirement already satisfied: numpy>=1.21.0 in ./venv/lib/python3.10/site-packages (from bitcoinlib) (1.23.1) Requirement already satisfied: requests>=2.25.0 in ./venv/lib/python3.10/site-packages (from bitcoinlib) (2.28.1) Requirement already satisfied: SQLAlchemy>=1.4.28 in ./venv/lib/python3.10/site-packages (from bitcoinlib) (1.4.39) Requirement already satisfied: pycryptodome>=3.14.1 in ./venv/lib/python3.10/site-packages (from bitcoinlib) (3.15.0) Collecting fastecdsa>=2.2.3   Using cached fastecdsa-2.2.3.tar.gz (48 kB)   Preparing metadata (setup.py) ... done Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./venv/lib/python3.10/site-packages (from requests>=2.25.0->bitcoinlib) (1.26.10) Requirement already satisfied: charset-normalizer<3,>=2 in ./venv/lib/python3.10/site-packages (from requests>=2.25.0->bitcoinlib) (2.1.0) Requirement already satisfied: certifi>=2017.4.17 in ./venv/lib/python3.10/site-packages (from requests>=2.25.0->bitcoinlib) (2022.6.15) Requirement already satisfied: idna<4,>=2.5 in ./venv/lib/python3.10/site-packages (from requests>=2.25.0->bitcoinlib) (3.3) Building wheels for collected packages: fastecdsa   Building wheel for fastecdsa (setup.py) ... error   ERROR: Command errored out with exit status 1:    command: /Users/ofekbenharosh/PycharmProjects/pythonProject/venv/bin/python -u
-c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/1s/0c6cnvt947n5r6_07k__th6r0000gn/T/pip-install-nf5yyqfc/fastecdsa_7d7c1187bcda423cbb18e2513ee27e01/setup.py'"'"';
__file__='"'"'/private/var/folders/1s/0c6cnvt947n5r6_07k__th6r0000gn/T/pip-install-nf5yyqfc/fastecdsa_7d7c1187bcda423cbb18e2513ee27e01/setup.py'"'"';f
= getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /private/var/folders/1s/0c6cnvt947n5r6_07k__th6r0000gn/T/pip-wheel-glm0yzzn
       cwd: /private/var/folders/1s/0c6cnvt947n5r6_07k__th6r0000gn/T/pip-install-nf5yyqfc/fastecdsa_7d7c1187bcda423cbb18e2513ee27e01/ Complete output (49 lines):   running bdist_wheel   running build   running build_py   creating build   creating build/lib.macosx-10.9-universal2-cpython-310   creating build/lib.macosx-10.9-universal2-cpython-310/fastecdsa   copying fastecdsa/benchmark.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa   copying fastecdsa/util.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa   copying fastecdsa/__init__.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa   copying fastecdsa/keys.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa   copying fastecdsa/curve.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa   copying fastecdsa/point.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa   copying fastecdsa/ecdsa.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa   creating build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/tests   copying fastecdsa/tests/test_point.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/tests   copying fastecdsa/tests/test_keygen.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/tests   copying fastecdsa/tests/test_nonce_generation.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/tests   copying fastecdsa/tests/test_key_recovery.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/tests   copying fastecdsa/tests/test_key_export_import.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/tests   copying fastecdsa/tests/test_whycheproof_vectors.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/tests   copying fastecdsa/tests/__init__.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/tests   copying fastecdsa/tests/test_prehashed.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/tests   copying fastecdsa/tests/test_brainpool_ecdh.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/tests   copying fastecdsa/tests/test_rfc6979_ecdsa.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/tests   copying fastecdsa/tests/test_whitespace_parsing.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/tests   copying fastecdsa/tests/test_prime_field_curve_math.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/tests   copying fastecdsa/tests/test_p256_ecdsa.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/tests   creating build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/encoding   copying fastecdsa/encoding/pem.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/encoding   copying fastecdsa/encoding/util.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/encoding   copying fastecdsa/encoding/__init__.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/encoding   copying fastecdsa/encoding/sec1.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/encoding   copying fastecdsa/encoding/der.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/encoding   copying fastecdsa/encoding/asn1.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/encoding   creating build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/tests/encoding  copying fastecdsa/tests/encoding/test_sec1.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/tests/encoding  copying fastecdsa/tests/encoding/__init__.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/tests/encoding  copying fastecdsa/tests/encoding/test_der.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/tests/encoding  running build_ext   building 'fastecdsa.curvemath' extension   creating build/temp.macosx-10.9-universal2-cpython-310   creating build/temp.macosx-10.9-universal2-cpython-310/src   clang
-Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch arm64 -arch x86_64 -g -stdlib=libc++ -Isrc/ -I/Users/ofekbenharosh/PycharmProjects/pythonProject/venv/include -I/Library/Frameworks/Python.framework/Versions/3.10/include/python3.10
-c src/curve.c -o build/temp.macosx-10.9-universal2-cpython-310/src/curve.o -O2   In file included from src/curve.c:1:   src/curve.h:4:10: fatal error: 'gmp.h' file not found   #include "gmp.h"
           ^~~~~~~   1 error generated.   error: command '/usr/bin/clang' failed with exit code 1
----------------------------------------   ERROR: Failed building wheel for fastecdsa   Running setup.py clean for fastecdsa Failed to build fastecdsa Installing collected packages: fastecdsa, bitcoinlib
    Running setup.py install for fastecdsa ... error
    ERROR: Command errored out with exit status 1:
     command: /Users/ofekbenharosh/PycharmProjects/pythonProject/venv/bin/python -u
-c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/1s/0c6cnvt947n5r6_07k__th6r0000gn/T/pip-install-nf5yyqfc/fastecdsa_7d7c1187bcda423cbb18e2513ee27e01/setup.py'"'"';
__file__='"'"'/private/var/folders/1s/0c6cnvt947n5r6_07k__th6r0000gn/T/pip-install-nf5yyqfc/fastecdsa_7d7c1187bcda423cbb18e2513ee27e01/setup.py'"'"';f
= getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/1s/0c6cnvt947n5r6_07k__th6r0000gn/T/pip-record-xuukb0iy/install-record.txt
--single-version-externally-managed --compile --install-headers /Users/ofekbenharosh/PycharmProjects/pythonProject/venv/include/site/python3.10/fastecdsa
         cwd: /private/var/folders/1s/0c6cnvt947n5r6_07k__th6r0000gn/T/pip-install-nf5yyqfc/fastecdsa_7d7c1187bcda423cbb18e2513ee27e01/
    Complete output (51 lines):
    running install
    /Users/ofekbenharosh/PycharmProjects/pythonProject/venv/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    running build
    running build_py
    creating build
    creating build/lib.macosx-10.9-universal2-cpython-310
    creating build/lib.macosx-10.9-universal2-cpython-310/fastecdsa
    copying fastecdsa/benchmark.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa
    copying fastecdsa/util.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa
    copying fastecdsa/__init__.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa
    copying fastecdsa/keys.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa
    copying fastecdsa/curve.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa
    copying fastecdsa/point.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa
    copying fastecdsa/ecdsa.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa
    creating build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/tests
    copying fastecdsa/tests/test_point.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/tests
    copying fastecdsa/tests/test_keygen.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/tests
    copying fastecdsa/tests/test_nonce_generation.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/tests
    copying fastecdsa/tests/test_key_recovery.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/tests
    copying fastecdsa/tests/test_key_export_import.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/tests
    copying fastecdsa/tests/test_whycheproof_vectors.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/tests
    copying fastecdsa/tests/__init__.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/tests
    copying fastecdsa/tests/test_prehashed.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/tests
    copying fastecdsa/tests/test_brainpool_ecdh.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/tests
    copying fastecdsa/tests/test_rfc6979_ecdsa.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/tests
    copying fastecdsa/tests/test_whitespace_parsing.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/tests
    copying fastecdsa/tests/test_prime_field_curve_math.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/tests
    copying fastecdsa/tests/test_p256_ecdsa.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/tests
    creating build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/encoding
    copying fastecdsa/encoding/pem.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/encoding
    copying fastecdsa/encoding/util.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/encoding
    copying fastecdsa/encoding/__init__.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/encoding
    copying fastecdsa/encoding/sec1.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/encoding
    copying fastecdsa/encoding/der.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/encoding
    copying fastecdsa/encoding/asn1.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/encoding
    creating build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/tests/encoding
    copying fastecdsa/tests/encoding/test_sec1.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/tests/encoding
    copying fastecdsa/tests/encoding/__init__.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/tests/encoding
    copying fastecdsa/tests/encoding/test_der.py -> build/lib.macosx-10.9-universal2-cpython-310/fastecdsa/tests/encoding
    running build_ext
    building 'fastecdsa.curvemath' extension
    creating build/temp.macosx-10.9-universal2-cpython-310
    creating build/temp.macosx-10.9-universal2-cpython-310/src
    clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch arm64 -arch x86_64 -g -stdlib=libc++ -Isrc/
-I/Users/ofekbenharosh/PycharmProjects/pythonProject/venv/include -I/Library/Frameworks/Python.framework/Versions/3.10/include/python3.10
-c src/curve.c -o build/temp.macosx-10.9-universal2-cpython-310/src/curve.o -O2
    In file included from src/curve.c:1:
    src/curve.h:4:10: fatal error: 'gmp.h' file not found
    #include "gmp.h"
             ^~~~~~~
    1 error generated.
    error: command '/usr/bin/clang' failed with exit code 1
    ---------------------------------------- ERROR: Command errored out with exit status 1: /Users/ofekbenharosh/PycharmProjects/pythonProject/venv/bin/python -u
-c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/1s/0c6cnvt947n5r6_07k__th6r0000gn/T/pip-install-nf5yyqfc/fastecdsa_7d7c1187bcda423cbb18e2513ee27e01/setup.py'"'"';
__file__='"'"'/private/var/folders/1s/0c6cnvt947n5r6_07k__th6r0000gn/T/pip-install-nf5yyqfc/fastecdsa_7d7c1187bcda423cbb18e2513ee27e01/setup.py'"'"';f
= getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/1s/0c6cnvt947n5r6_07k__th6r0000gn/T/pip-record-xuukb0iy/install-record.txt
--single-version-externally-managed --compile --install-headers /Users/ofekbenharosh/PycharmProjects/pythonProject/venv/include/site/python3.10/fastecdsa Check the logs for full command output.

The bitcoinlib package depends on fastecdsa which requires (the gmp.h from the) GNU Multiple Precision Arithmetic Library (GMP), so install that before installing bitcoinlib e.g.
sudo apt-get install libgmp-dev


text:
How do we know a new transaction is not corrupted/manipulated?
let's say I want to send a certain amount of bitcoin to a certain client, but my computer is connected to a public wifi, how do I make sure there is not a man in the middle altering the transaction broadcast?
As far as I know, the underlying mechanisms to broadcast operations are not encrypted, how does the network recognize corrupted packets?
Moreover, could a government analyze a traffic and "block" packets that gives hint that a specific packet contains bitcoin-protocol-relevant payload?

Yes, all traffic on the Bitcoin network is unencrypted, so it's pretty obvious when you're using Bitcoin.

@Murch But couldnt governments exploit this fact by blocking packages that contain btc-protocol-specific payloads, thus prohibiting bitcoin transaction within the state?

They could block unencrypted Bitcoin traffic but that wouldn't prevent Bitcoin transactions in their territory. It's not exactly hard to tunnel traffic through a VPN, or onto a privacy network. Also, https://www.bip324.com/.


text:
How are crypto prices calculated by an exchange? Example game in a spreadsheet to illustrate
I find the mechanisms behind crypto fascinating, so I wanted to do the thought exercise of creating my own coin, and seeing how it evolves in a simplistic spreadsheet-based example such as this one:

The idea is that each month, in turn, each person (that would be me, here, every time) decides whether to buy, or sell, "Crypto" coins, and how any of them. Each person is different in the amount they already own (the creator of the coin) or the amount of money they have at their disposal.
I know that when creating a coin, the creator determines: a) how many coins it mints, b) how many coins the creator keeps, c) how many coins are injected as liquidity in the exchange, and d) for how much money when adding in the liquidity to set the initial price. The table is here populated with arbitrary data. I'm assuming here that this coin has a fixed supply and no more can be minted, to simplify things.
... Happy as a clam, I tried to run "month 2" - then I realised, that someone who believes in this coin would buy a lot of it because it's cheap, effectively all that's available as a liquidity seeing how cheap it is. What now? If nobody sells, does the exchange automatically increase the price until holders are incentivised to sell? Second, if for some obscure reason here I would not buy all the liquidity, would the price increase because someone bought it? By how much?
This made me realise that I have absolutely no clue about how exchanges calculate the price of a coin in their system. Is it always the same formula (and just the liquidity + members change) or does each exchange have their own algorithm? How could I simulate an exchange here?
This would help me tremendously if someone could actually try to "play" this example in their answer to illustrate. I'm sure this will be of interest to many as well, because I found this question around quite a lot in my research, but never has how been answered.

This is a question on how markets price assets, and not really about bitcoin. Just imagine if you are in a room with 100 people and you wanted to sell some bitcoin, you could "ask" for $1 million per coin, but out of the other 99 people none of them want to pay $1 million per coin, so they put in their "bid". Eventually someone will offer to sell at a price that a buyer agrees to and you have the asset price. Some markets with low volume have very wide bid/ask spreads because there aren't enough people to provide a fair price, so many times larger exchanges have the "fairest" price.

Thanks, this is insightful. First, is there a stackexchange which is better suited to this question? I thought this was it because it applies to bitcoin as well (just replace coin with bitcoin in my text). Second, in my very limited experience of using exchanges (Binance), I don't really have a say in the ask/bid, the price is given to me. Assuming even I can set my own ask/bid prices, how is this price tag calculated?

Also, unless I'm mistaken, I still can't compute what would happen in the "what now?" case explained above. Would you care to give a possible solution?

Does this answer your question? How does an exchange calculate the price of a coin? ... Also see https://bitcoin.stackexchange.com/search?q=exchange+prices+calculate

@RedGrittyBrick partly yes, thanks - sorry, I had looked up previous posts and for some reason did not see this exact match (shame on me). I still don't know exactly how liquidity intervenes in there though: if it's only a bid/ask match, what's the point of liquidity in the exchange which is supposed to smooth out price variations?

I'd say the effect of liquidity is indirect, just as general market mood has indirect effects. Neither of them are used directly or explicitly to set a market price but both of them affect prices in one way or another. You can measure actual prices paid without needing to measure liquidity or mood.

Exchanges don't calculate prices, they report them. They aggregate people who want to sell and people who want to buy and report the prices they are willing to buy and sell for. When one person is willing to sell at a price that another person is willing to buy, they facilitate that exchange and report the price.
Say you look at what's going on at a particular exchange with a particular asset. There cannot exist a person who is willing to buy that asset at a price another person is willing to sell it for. Why? Because if that were the case, they would have already transacted.
So there must be a highest price anyone is willing to buy the asset for and a lowest price anyone is wiling to sell the asset for. And the highest buy price must be lower than the highest sell price.
So may there are people willing to buy ABC for $100 or less and people willing to sell ABC for $101 or more. At this exchange, you can now buy ABC for $101 or sell ABC for $100.
This can happen for some time, with people buying ABC at $101 and selling ABC at $100. The trades take place at different prices (buys at $101, sells at $100) but the price isn't actually changing. (This is why even though every transaction is both a buy and a sell since someone is selling and someone is buying, a "buy" is still a logically different transaction from a "sell" at an exchange.)
If someone buys all the ABC that people are willing to sell for $100, then anyone who wants to buy ABC will have to pay more than $100 for it. That's how prices go up.
Similarly, if sells exhaust all the willing buyers at $100 by selling more ABC than that, the next sales will have to go to buyers willing to pay less than $100. That's how prices go down.

Well put, thank you (+1). A couple of clarifications before I accept the answer: 1/ Does that mean that when Binance or other exchanges force le to buy a certain amount of coins for the inputted value of exchanged currency, it's really a snapshot at a given time of the lowest price I can buy this amount of coins in the exchange at this time? 2/ Can you confirm this means the buy and sell prices shown on exchanges can be very different and only depend on respectively the sellers' and buyers' offers in the exchange? 3/ What's the point of liquidity in the exchange's pool then?

Yes. If you buy more than is available at the stated price, you will pay more. 2. Yes, but in practice they will be very close in efficient markets. 3. Those people willing to buy and sell are liquidity in the exchange's pool. They're what lets you go to the exchange and buy and sell because there are people already willing to sell and buy.

Excellent, thanks. I had it all wrong. I accepted the answer, thanks. As a bonus: when you create a new coin, does that mean that you basically first give some for free to the exchange itself, and buy it back for a price which is automatically accepted by the exchange in order to set the initial price reported?

Update: again, I was wrong. Looks like when adding liquidity, we're actually staking a pair, so both the token and its exchange currency - the ratio sets the price of the token. I wonder if the exchange automatically accepts a buy request up to the number staked at the reported price? I assume that if we can, as soon as we siphon the exchange, the price will go up if someone else wants to buy and offers a buy request for a higher price.

@MisterMystère Exchanges don't own tokens, they just bring buyers and sellers together and hold tokens owner by others to facilitate staking and trading. When a token changes hands between someone who wants to buy it and someone who wants to sell it, exchanges report the transaction, including the price. Someone placing a buy request for a higher price will not change the price unless nobody was willing to sell at less than that price.

@MisterMystère Say I want to buy one bitcoin and I'm willing to pay up to $40,000 if I have to. That won't change the price of bitcoin to $40,000 since there are still plenty of people willing to sell for less than that and I won't have to pay $40,000. The price of bitcoin only goes up to $40,000 if nobody is willing to sell for less than that.


text:
Should boost be installed in /usr/ or /usr/local/ for bitcoin-core?
When installing bitcoin-core, libbitcoin or libbitcoin-explorer the build requires the C++ boost library. Is there any reason why I should prefer to install boost in the /usr/ hierarchy rather than /usr/local. Or should I not attempt to link all my C++ projects to the same boost build anyway, and link to a seperate boost build (of a possibly different versions installed in different locations) for each project?

Libraries should be installed in /usr/include.
For Boost, the path to it should be /usr/include/boost. That's where the linker would be looking for the libraries.
Otherwise, you may elect to install the libraries in some different directory and symlink it to /usr/include anyways.

Thank you! it seems however that not everyone agrees (e.g. https://ubuntuforums.org/showthread.php?t=1528661). I now have the impression that unless your boost library is coming from a distribution package, it is advised to stick it in /usr/local/include/boost and /usr/local/lib  rather than in the /usr/ hierarchy. I'll wait a bit before accepting this answer as the definite truth :)

This seems like something more suitable at unix stack exchange if we really wanna discuss where is best to install any arbitrary Libs.


text:
Java bitcoinj: Failure of getPrivateKeyEncoded or getKey?
I have the feeling something is not right with the method getPrivateKeyEncoded (which returns a DumpedPrivateKey object) of the ECKey class. If I attempt to recover the orginal key from the DumpedPrivateKey object using the getKey method, the retrieved key appears to be wrong (in the compressed case). I would be grateful if someone could tell me whether I am missing something, or whether this is a deliberate API choice etc. I attach a sample Test program to illustrate the issue:
// running bitcoinj-core-0.14.3.jar
import java.math.BigInteger;
import org.bitcoinj.core.NetworkParameters;
import org.bitcoinj.core.ECKey;
import org.bitcoinj.core.DumpedPrivateKey;

public class Test {

  public static void checkEquals(BigInteger n1, BigInteger n2, String msg){
    if(!n1.equals(n2)){
      System.err.println("failure in " + msg + ":");
      System.err.println("n1 = " + n1.toString(16));
      System.err.println("n2 = " + n2.toString(16));
      System.exit(1);
    }
  }

  public static void main(String[] args){

    String mainAsString = NetworkParameters.ID_MAINNET;
    String testAsString = NetworkParameters.ID_REGTEST;

    NetworkParameters mainNet  = NetworkParameters.fromID(mainAsString);
    NetworkParameters testNet  = NetworkParameters.fromID(testAsString);

    // compressed key
    ECKey k1 = new ECKey();

    // same key uncompressed
    ECKey k2 = k1.decompress();

    // common private secret
    BigInteger secret = k1.getPrivKey();  // same as k2.getPrivKey()

    // 4 possible types of WIF
    String wifMain1 = k1.getPrivateKeyAsWiF(mainNet); // main, compressed
    String wifMain2 = k2.getPrivateKeyAsWiF(mainNet); // main, uncompressed
    String wifTest1 = k1.getPrivateKeyAsWiF(testNet); // test, compressed
    String wifTest2 = k2.getPrivateKeyAsWiF(testNet); // test, uncompressed

    // creating corresponding DumpedPrivateKey
    DumpedPrivateKey dpkMain1 = DumpedPrivateKey.fromBase58(null, wifMain1);
    DumpedPrivateKey dpkMain2 = DumpedPrivateKey.fromBase58(null, wifMain2);
    DumpedPrivateKey dpkTest1 = DumpedPrivateKey.fromBase58(null, wifTest1);
    DumpedPrivateKey dpkTest2 = DumpedPrivateKey.fromBase58(null, wifTest2);

    // reconstructing ECKey from DumpedPrivateKey
    ECKey keyMain1 = dpkMain1.getKey();
    ECKey keyMain2 = dpkMain2.getKey();
    ECKey keyTest1 = dpkTest1.getKey();
    ECKey keyTest2 = dpkTest2.getKey();

    // Testing all is fine (should also check compression status)
    checkEquals(secret, keyMain1.getPrivKey(), "test1");
    checkEquals(secret, keyMain2.getPrivKey(), "test2");
    checkEquals(secret, keyTest1.getPrivKey(), "test3");
    checkEquals(secret, keyTest2.getPrivKey(), "test4");

    // so far so good...

    // now creating DumpedPrivateKey objects from getPrivateKeyEncoded
    dpkMain1 = k1.getPrivateKeyEncoded(mainNet);
    dpkMain2 = k2.getPrivateKeyEncoded(mainNet);
    dpkTest1 = k1.getPrivateKeyEncoded(testNet);
    dpkTest2 = k2.getPrivateKeyEncoded(testNet);

    // again reconstructing ECKey from DumpedPrivateKey
    keyMain1 = dpkMain1.getKey();
    keyMain2 = dpkMain2.getKey();
    keyTest1 = dpkTest1.getKey();
    keyTest2 = dpkTest2.getKey();

    // testing is successful for uncompressed keys ...
    checkEquals(secret, keyMain2.getPrivKey(), "test6");
    checkEquals(secret, keyTest2.getPrivKey(), "test7");

    // ... but fails for compressed keys
    checkEquals(secret, keyMain1.getPrivKey(), "test8");
    checkEquals(secret, keyTest1.getPrivKey(), "test9");

    // the output shows a trailing suffix byte 0x01
    /*
    failure in test8:
    n1 = 3e27a3956811801724857245ae78dd8ba1ce3df60da2bb00a782cee66ed640ee
    n2 = 3e27a3956811801724857245ae78dd8ba1ce3df60da2bb00a782cee66ed640ee01
    */
  }
}


text:
What is the NAT traversal technique used by bitcoin
https://bitcointalk.org/index.php?topic=30947.msg23736281#msg23736281 says that

incoming port can be opened or
UPnP enabled.

Is this incoming port opened on the NAT device so that all incoming messages coming to that port are relayed to my host?
EDIT: More resources about the topic https://bitcoin.org/en/full-node#enabling-connections and https://bitcoin.org/en/full-node#network-configuration
The second resource says: "If Bitcoin Core can’t automatically configure your router to open port 8333, you will need to manually configure your router."
Is this automatic opening mentioned actually UPnP way?
And manually configuring the router actually bullet 1 mentioned above?

Yes, that's what it's talking about, but it isn't something that Bitcoin Core does itself.  The post is just saying that you should manually configure your NAT device to do that.

Is this automatic opening mentioned actually UPnP way?

Yes. UPnP is disabled by default in Bitcoin Core due to some vulnerabilities in UPnP. You can enable it by going to Settings > Options > Network and checking Map port using UPnP

And manually configuring the router actually bullet 1 mentioned above?

Yes.


text:
Create Custom Script with 2-3
I'm studying P2SH, and I want to create a custom script.
I already created one tx but without OP_CHECKMULITISIG as you can check at this link https://tbtc.bitaps.com/c86e443bdb3ad4845b1670a63376daf785eae3932a1b6146a7138ea46d51d24a.
The script is very simple, in Scriptpubkey you can find
OP_3 OP_EQUAL

And in ScriptSig I put
OP_3

The whole script is:
02000000014ad2516da48e13a746611b2a93e3ea85f7da7633a670165b84d43adb3b446ec8000000000453025387ffffffff029f0f0000000000001976a914261216d05727044e927c385aa9bd278754cdc8dc88ac0000000000000000136a11636f72736f626974636f696e2e636f6d0a00000000

Now, I want to add 2-3 into simple script.
My redeem script is:
53875221"$PB1"21"$PB2"21"$PB3"53AE

where $PB1 $PB2 $PB3 are compressed public keys.
Then I try to sign the transaction and add OP_3 in ScriptSig (increase its length by 1 byte)
Original Transaction Data

02000000014218ce881112706b0736419eef81220255d8dcc1412233a47091b770534f9b47000000006d4c6b538752210319ea83d5ab6d737133c6dad6a4f942a87d3f0e7b6ad4050d245c34103319a1c3210211ebc7feb791c8936c7a8d3661bbb54470ecf579fd008d279cda67fc3e7374c721023dfba960bd3b37e004e34cd7777b9b402fb95d145d2649c3adfd5a93411c2c3453aeffffffff025044042a010000001976a91475704b43283de0bdca419b898250e1916acc9df088ac0000000000000000136a11636f72736f626974636f696e2e636f6d0a00000000

Transaction data with OP_3 in scriptSig

02000000014218ce881112706b0736419eef81220255d8dcc1412233a47091b770534f9b47000000006e534c6b538752210319ea83d5ab6d737133c6dad6a4f942a87d3f0e7b6ad4050d245c34103319a1c3210211ebc7feb791c8936c7a8d3661bbb54470ecf579fd008d279cda67fc3e7374c721023dfba960bd3b37e004e34cd7777b9b402fb95d145d2649c3adfd5a93411c2c3453aeffffffff025044042a010000001976a91475704b43283de0bdca419b898250e1916acc9df088ac0000000000000000136a11636f72736f626974636f696e2e636f6d0a00000000

When I send the transaction I get this error:

error code: -26 error message: mandatory-script-verify-flag-failed
  (Operation not valid with the current stack size) (code 16)

How did you try to sign your transaction? There are no signatures in your transaction, so the OP_CHECKMULTISIG is failing.
Because you are using a custom script, signing software are unlikely to be able to sign your transaction because they can only sign for standard templates. You are not using a standard script template (because of the OP_3 OP_EQUAL) so no signatures are being created. To sign this transaction, you will need to produce the signatures manually (by computing the correct sighash and signing it) and insert them into the transaction manually.

Thanks for reply. I'm using Bitcoin core and I'm using this "flow" to understand how it works. I create an address P2SH with OP_3 OP_EQUAL and 3 compressed public keys.  After that I create a transaction with createrawtransaction like bitcoin-cli createrawtransaction '[{"txid":"'$TXID'","vout":'$VOUT',"scriptPubKey":"'$SCRIPTPUBKEY'","redeemScript":"'$REDEEM'"}]' '[{"'$ADDR_MITT'":'$AMOUNT'}'  and try to sign with bitcoin-cli signrawtransactionwithkey $TX_DATA '["'$PK1'","'$PK2'"]' '[{"txid":"'$TXID'","vout":'$VOUT',"scriptPubKey":"'$SCRIPTPUBKEY'","redeemScript":"'$REDEEM'"}]'

Bitcoin Core doesn't know your script so it can't produce signatures for it.

When you say, "you will need to produce the signatures manually" what does mean? With openssl for example? With another RPC? thanks again

You need to use some other software that can produce Bitcoin signatures and then insert those signatures into your transaction by hand.