text:
Bitcoind does not like ECDSA (r, s) pair produced by OpenSSL
I am writing transactions manually and have stumbled across a rather bizarre situation.
Only one in a few of the transactions I broadcast to bitcoind is accepted, otherwise I get a REJECT_NONSTANDARD (Non-canonical DER signature).
So I got my hands dirty and tracked the rejection to be originating from this line: https://github.com/bitcoin/bitcoin/blob/9c5f0d542d1db507b3b9c87bd9de6d0d758d51c1/src/script/interpreter.cpp#L163
I read about DER encoding and checked how IsValidSignatureEncoding is enforcing it, but I do not know why OpenSSL generates not-DER-compliant (r, s) values?
How should I overcome this? I am thinking of something along the lines of (pseudocode):
Pair (r, s);
do
{
   (r, s) = sign(hash, pvtkey);
} while (r[0] >= 128 || s[0] >= 128); // where r[0], s[0] should be the very first byte of each value

But isn't that kind of redundant? Can I give OpenSSL any flag to produce a valid DER (R, S) pair in the first place?

Are you using OpenSSL to encode the DER or did you write your own DER encoding function? It's very easy to end up with a function that is only correct for some inputs.

A possible reason is that Bitcoin Core expects a low S value. Try changing s with N-s if s > N/2 (N is the curve order).
Source here

Actually that was not my initial issue, that's a completely different error code which I am getting now (SCRIPT_ERR_SIG_HIGH_S) after prepending my faulty S value with an 0x00 byte (I read in some other thread that should be done if the initial S's first byte is >= 128). So yeah most likely what I have to do now is what you suggested

The following python code can create a valid DER encoded signature given r and s as byte objects:
def ser_sig_der(r, s):
    sig = b"\x30"

    # Make r and s as short as possible
    ri = 0
    for b in r:
        if b == "\x00":
            ri += 1
        else:
            break
    r = r[ri:]
    si = 0
    for b in s:
        if b == "\x00":
            si += 1
        else:
            break;
    s = s[si:]

    # Make positive of neg
    first = r[0]
    if first & (1 << 7) != 0:
        r = b"\x00" + r
    first = s[0]
    if first & (1 << 7) != 0:
        s = b"\x00" + s

    # Write total length
    total_len = len(r) + len(s) + 4
    sig += struct.pack("B", total_len)

    # write r
    sig += b"\x02"
    sig += struct.pack("B", len(r))
    sig += r

    # write s
    sig += b"\x02"
    sig += struct.pack("B", len(s))
    sig += s
    return sig

It is important to note that signatures in Bitcoin also contain an extra byte appended to the DER encoded signature which represents the sighash type. You will need to add that byte yourself.

The third byte encodes information about the content length.
If content length is less than 127 bytes third byte is equal to content length.
If it is greater than 127 and less than 255, the third byte is 0x81.
If it is greater than 255, the third byte is 0x82.
The next byte(s) indicates the content length.
Reference:
https://github.com/openssl/openssl/blob/master/crypto/asn1_dsa.c  encode_der_length
https://www.itu.int/rec/T-REC-X.690-201508-I/en

Although bitcoin signatures use secp256k1, and low-S as MikeD noted, so all of the value parts never exceed 69 bytes and your second and third cases of length encoding never occur. (The third case actually is >=256 and <65536; for longer than that, there are more length encodings you don't mention, but they aren't supported or needed by the code you link.)


text:
How do I search for a hidden and disguised (extension changed) wallet file on damaged hard drive?
Back in 2012 we had a bitcoin or two on a computer that crashed. It was a dual boot system and the wallet was in the Linux partitions. The user, a family member, changed the file extension to a extension document, probably open office or abiword. I am trying to reconstruct the hard drive, and have recovered the files from the Linux partitions. I need to know how to search for and find the file now. I have been researching my butt off trying to find a reliable way to find it. The only computer I have right now is an old XP laptop with 512 megs of RAM, won’t have a newer computer for another 10-15 days (got an excellent dell direct deal but shipping and delivery is absurdly slow). I have done plenty of desktop support and home computer repair for a living, but have been retired since 2014 and my Data recovery experience is limited.
What I need most right now is a method to identify which among the hundred plus gigs of files is the wallet. First question is what is the minimum size of a wallet.dat file, so I can limit the time searches take by ignoring the zillions of files too small to be a wallet. I have seen one suggestion that new wallet files are about 96k.
Second question is how do I search for it? I need a search term, a unique string of text or characters that is present inside of every bitcoin wallet. And will Windows file manager find the text inside of the wallet, or will I need yet another tool? I have been using a string of numbers (62 31 05 00) that was suggested on a post here but have no idea how effective that might be.
Please note I have no Linux system or live cd for using Linux, and zero experience with Linux   I don’t think this is the time for me to master the Linux command line, the potential for screwing up and destroying data is too high.  I have imaged the Linux partitions from the old drive onto a 2 terabyte usb drive, and have recovered the files to that drive. Took me over a week. So the files are on a NTFS usb drive and should be safe to explore using Windows tools.
Any help will be greatly appreciated.

Use the unix tool file to search for the magic number of the wallet.dat. Some shell script to loop over all files should have it in minutes if the filesystem hasn't been damaged. https://en.wikipedia.org/wiki/Magic_number_(programming)#Magic_numbers_in_files Edit: It will be much easier from a linux system. I wouldn't even dare to mount this disk in windows for being too afraid of windows overwriting some stuff on the linux filesystem.

I should have been more detailed in my original post regarding the tools I have available and my capabilities. I have no Linux system or live cd for using Linux, and zero experience with Linux   I don’t think this is the time for me to master the Linux command line, the potential for screwing up and destroying data is too high.  I have imaged the Linux partitions from the old drive onto a 2 terabyte usb drive, and have recovered the files to that drive. Took me over a week. So the files are on a NTFS usb drive and should be safe to explore using Windows tools.

I'm voting to close this question as off-topic because this question is not bitcoin related. It is file system related, and having zero knowledge of Linux, and wanting to search from Windows goes far beyond scope of forum here.

I understand your confusion, but I am not asking for assistance in searching for the files. Or with Linux, as the files are on a NTFS volume on a usb external drive, attached to a Windows machine. The Linux info is useless to me, and bitcoin is not Linux only anyway. I still need keywords that only a bitcoin techie would know (or be able to find by opening a new and unused bitcoin wallet)), and what size a new bitcoin wallet from 2012 was. I have no idea why anyone wants to try to help me with using Windows or the Windows tools; I do not want or need that at all.

Try searching words fromaccount or timesmart. Wallet.dat is binary, so there should'nt be too many words accessible, but these should.

No luck. I may try downloading the bitcoin core from 2012 and opening the wallet that comes with it in notepad, that will show me what windows sees for code and other strings inside the wallet. Then search for some of those strings. And/or trying using Agent Ransack to search in Windows, as Windows explorer isn’t so hot.

Where this file is should be easy to find if it was only an extension rename.
If you recovered the directory structure also check the default Linux directory.
~/.bitcoin/
The original wallet.dat will be in the parent directory of the blockchain files seach for "blk00000.dat" to find that dir.
If just the extention was changed as you said, index the drive and search for wallet*.* using Windows explorer.
The wallet.dat file shouldn't be very large, only 1,390 kb for a wallet with 4 keys or so.
If the directory structure was lost and you have all these files in one directory. You can try to use Windows indexing further.
Rename all the recovered files to have a .txt extention.
From cmd window "ren *.* *.*.txt" that will preserve the old extention but also give it a .txt extention like "notepad.exe.txt"
Then you can allow Windows to index all the content easily. And here's the gold. Search for "keymeta!" inside the files, your wallet.dat will have 100s of uses of this string.
Hope this helps. If you cash in share the wealth. Tips appreciated. :-)
13aRZ2u3Wzim5vjDpVaxQFP4v6ESn5hH5E

I’m pretty sure the wallet file was moved, it certainly isn’t named wallet or .dat as the user was trying to hide it and renamed it. It’s only one bitcoin and in 2012 was not a big amount. But the “keymetal” is golden for me, and the size helps a lot too, as there’s a huge amount of files under 1,000 kb in over 450 gigs. Thank you very much!

I would definitely narrow the file size to search to 1k to 25k size. Also a lot of that file space may be the blockchain files, blk*.dat files of 131mb each in 2012 probably about 5g worth. Those could be moved out of the search. What ever you do, if you decide to bulk rename / delete etc. Back up first.

I’ve only been searching for files over 50k, I saw in my research that an empty wallet was over 96k. I will run those again to include 1k and larger. There were 10 bitcoins originally and even with only one left I have the impression that wallets get larger with every transaction so I will set the upper limit mit high. I usually start the searches just before bed as they are long and the computer doesn’t  care if I am asleep. I am going to find that wallet! Thanks again.

@Kilroy indeed a wallet with a lot of transactions can get to a few mb. But a new wallet is only 1kb or so.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/

sort: votes


text:
How to hold a fiduciary accountable?
Fiduciaries (executors, trustees, guardians) hold and manage funds on behalf of others (heirs, beneficiaries, etc). So it's important to be able to audit a fiduciary's transactions, to ensure he hasn't mismanaged or stolen any funds.
With bank and brokerage accounts, this can be pieced together from account statements. How can a bitcoin wallet be similarly audited?
Simplified Example: Upon death, the executor takes possession of the deceased's hardware wallet. Over the course the probate process, executor spends to pay creditors, taxes, etc.
At the end of probate, executor proposes to distribute the remaining bitcoin/utxos equally to the heirs, as per the will.
Had this been bank/brokerage accounts, the Executor would use statements to create a paper trail of all the dollars he collected, what he spent, and thus the remaining balance for the heirs.

But how can the heirs confirm what was the "opening balance" of the wallet? The executor could show a list of addresses and utxos when he took possession of the wallet, but how could anyone verify that it's a complete list? Would sharing the wallet xpub address be sufficient to show heirs (and protect the executor)?

If there's a satisfactory answer to this, the rest (auditing executors's spending, remaining balance, etc.) seems like basic on-chain tracking, assuming he spends on-chain and not lightning. Am I wrong?

Thank you
EDITED: Added possibility of xpub address

Traditionally there is no definitive way to determine which addresses belong to which wallets, so you can't know the opening balance for sure. However..
In the case that the wallet is a hierarchical deterministic wallet and the seed data is revealed, you can generate all the keys associated with that wallet and scan the blockchain for all UTXOs associated with any of those keys.
However, whether or not there exist other keys on the wallet that are not associated with the HD seed phrase is still unknown. (depends if the particular wallet functionality allows both HD and non-HD on one wallet, I guess)
The best recourse here would be to use data analysis tricks to try and link addresses through change transactions, etc. But if the fraudulent executor is clever they can easily cover their tracks.

What if the Executor shared the wallet xpub address with the heirs (or the court, as needed), essentially offering a watch-only version of the wallet to anyone with the right to audit his executor role?

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Is the "BIP39 words contain all the bitcoin in the world" meme correct?
A common bitcoin meme is to post an image of the BIP39 wordlist with a caption "Here is access to all of the bitcoin in the world, you’re welcome.".
Is this strictly true? Does the set of all 12+24 permutations of the BIP39 words generate all possible valid private keys?

It is not possible to know whether all private keys can be generated with BIP 39 as the only way to do that would be to generate all possible mnemonics and record what private keys they produce. The number of mnemonics and the number of private keys is so large that this is impossible to do - that is what makes them secure.
Otherwise, we do not know for sure whether the private keys generated are uniformly random. However, it's a reasonable assumption that they are. The hash functions in use, although they have the same issue of not knowing for certain, are widely used and believed to be uniformly random. If this assumption is true, then the private keys generated should be uniformly random as well, which means that BIP 39 should produce every possible private key.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Do know which UTXO fees are chosen from? Need help on an inscription in a native segwit wallet
I have a UTXO with an inscription that is now valuable, but I accidentally received it into a non-Taproot (bech32, native segwit) wallet.
I'm now aware that because of this, I can't use Ordinal theory to control which satoshis are being spent when I send this transaction.
What I'm wondering is, is there any way to dictate which UTXO the fees come from? This was accepted into a clean wallet, so it's currently the only UTXO. If I add another UTXO, would the fees come from that instead? Thus ensuring that the first UTXO is fully received by the receiver.
I'm aware of one person who was in the same situation. He sent two more UTXOs of 100k satoshis into the same wallet, and then paid the minimum possible fee while keeping bytes low, then pushed his transaction through the VIA btc transaction accelerator. The fee was about ~250 sats, so he had a ~250/200000 chance of failure. It went through successfully.
I'm mostly curious if there's anything else I can do to increase my odds here, and if the logic of what I was told above is sound. It seems obvious that the sats I'm sending are the same ones that are being received by the receiver, they're not mixed within blocks, and that the only risk is that my inscription happens to be attached to one of the satoshi's within the fee. I'm also curious if miners could "snipe" the inscription, but I'm assuming that's not possible as well.

What wallet are you using? A wallet with coin control should let you pick what utxos to spend from and where the fee should be deducted from

is there any way to dictate which UTXO the fees come from?

Some wallets have a feature named "coin control".
The fee is simply the difference between input amount and output amount. There is no correspondence between individual inputs and individual outputs when there are multiple of either - the notion of fees coming from specifically one of several inputs simply has no meaning in the Bitcoin network.
The Ordinals project makes an arbitrary assignment according to its own off-chain rules. This is based on the ordering of inputs and outputs in the transaction - but this order has no meaning in the Bitcoin network and can be random. I guess you would have to manually construct a transaction that makes this order correspond to the external Ordinals rules you want to apply.

According to the Ordinals FAQ, the ordinals are assigned first-in-first-out. Each satoshi in the inputs is assigned in order to each first available satoshi in the outputs. Any satoshis from the inputs not assigned to outputs are assigned to the fees after that.
For an example, let’s assume a transaction has two inputs of three and four sats, two outputs of four and two sats, and one sat going to fee. Let’s assume the ordinals of the input sats are [e f g] and [a b c d].
 Inputs         Outputs

[e f g]     ↦  [e f g a]
[a b c d]      [b c]
                            last sats of last input go to fee ⇒ [d]


text:
Can anyone explain this vulnerability in PPC
Can anyone explain the vulnerability of PPCoin and possibly other Proof of Stake coins that might allow an attacker to double spend? I've seen one explanation here:
https://bitcointalk.org/index.php?topic=169204.msg1761154#msg1761154
but it wasn't clear.

PPCoin and other coins use Proof of Stake in addition to Proof of Work.  Stake is how long a coin has been idle multiplied by its value.  If you've been holding many coins for a long time, you can spend their coin age in lieu of poof of work.  The fear is that someone will hoard coins, gather a bunch of age, and then double-spend and use his accrued coin age to give each spend 6 confirmations immediately, without needing CPU work.
It's a valid concern, in my opinion.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
How to generate a bitcoin public key from an existing private key using openssl?
I want to generate bitcoin priv/pub key manually.
I generated random binary numbers by flipping some coins and converted them to hex.
Now,I got a bitcoin private key
in hex:d1267088f0defab13969dd34c899d28268119cf9e68f3856d0ee9b581f5a1bca
in WIF:L4EGhR5Fdon96W2zepTAE3CxtayBqiN3v2pBmkB9dxG4r8JaVKf2
how to gerate Public Key using openssl command line in linux?
I tried to use
openssl ec -in priv_key.pem -pubout

but terminal said: unable to load Key, and lots error file uris.
Maby I got something wrong with PEM private key format.
The priv_key.pem ,I converted private key hex to base64 format.
-----BEGIN EC PRIVATE KEY-----
ZDEyNjcwODhmMGRlZmFiMTM5NjlkZDM0Yzg5OWQyODI2ODExOWNmOWU2OGYzODU2
ZDBlZTliNTgxZjVhMWJjYQ==
-----END EC PRIVATE KEY-----

The SECG format for EC privatekey used by OpenSSL is quite different from just the raw bits/number 'd' used in Bitcoin. See crossdupe https://stackoverflow.com/questions/48101258/how-to-convert-an-ecdsa-key-to-pem-format -- where my answer is based on https://bitcoin.stackexchange.com/questions/66594/ but that requires both privatekey and publickey and you apparently don't have publickey (yet; my answer derives it).

OpenSSL will compute a public key corresponding to a private key, try:
$ openssl ec -in private_key.openssl.pem -text

There must be correct input provided to OpenSSL. It is not enough to convert private key from hex string to base64 in order for it to be accepted by OpenSSL. There must be eliptic curve parameters included in the .pem file, like shown in the short script:

https://gist.github.com/vkobel/52048e12d8930ef63930949a717ddcbf


text:
Bitcoin Core missing public key when generating PSBT transaction
I use Bitcoin Core for offline signing. I created a legacy watch-only wallet, imported a public key using importpubkey, then called rescanblockchain, then generated a psbt transaction text(psbt_tx_txt), and finally successfully signed offline with my Bitcoin Core signing client.
However, if I restart the Bitcoin Core watch-only client, and regenerate psbt_tx_txt, it will missing the public key: analyzing psbt_tx_txt with analyzepsbt, it prompts missing pubkey:
{
  "inputs": [
    {
      "has_utxo": true,
      "is_final": false,
      "next": "updater",
      "missing": {
        "pubkeys": [
          "a_public_key_id"
        ]
      }
    }
  ],
  "fee": 0.00001100,
  "next": "updater"
}

But when I query the address information with getaddressinfo, the public key is exactly included in it:
{
  "address": "bc1......",
  "scriptPubKey": "......",
  "ismine": false,
  "solvable": true,
  "desc": "wpkh(......)#l8qyd3u4",
  "iswatchonly": true,
  "isscript": false,
  "iswitness": true,
  "witness_version": 0,
  "witness_program": "......",
  "pubkey": "a_PUBLIC_KEY",
  "ischange": false,
  "timestamp": 1,
  "labels": [
    "......"
  ]
}

If I use the psbt_tx_txt to sign on the signing client, the bitcoin core client prompts: transaction is missing some information about inputs.

After restarting Bitcoin Core, it doesn't seem to be able to find the previously imported public key.
This means that every time I create an offline signed transaction, I need to create a new wallet, importpubkey,and rescanblockchain.
I think the public key should be able to be inserted into the psbt_tx_txt, next time I can JUST create the psbt, am I missing something?

[update]
The issue can be reproduced by this:
Starting the bitcoin-qt GUI client and synchronizing the blocks. Create a blank ledgcy watch-only wallet and enable the coin control feature in the bitcoin-qt preferences settings.
Import the public key of the address with the balance using importpubkey in the bitcoin-qt console.  Select the Send main menu, select an input in the inputs option, go back to the Send main menu and leave the change address empty. Fill in a receiving address, select the maximum amount(Use available balance), enable Subtract fee from amount, customize the transaction fee to a fixed value, disable  Replace-By-Fee feature, click Create Unsigned button to create psbt-string-A.
Restart bitcoin-qt, select the wallet you just created, and repeat the the same operation to create psbt-string-B.
Analyzing psbt-string-A and psbt-string-B with analyzepsbt in bitcoin-qt console, psbt-string-A is normal, psbt-string-B is missing public key.

After you had signed the psbt, did you broadcast the transaction? Are you giving your watch-only wallet exactly the same psbt?

@AndrewChow Yes, after signing the psbt, I broadcast the transaction. Yes, it was exactly the same psbt.

To clarify, you used exactly the same psbt string? You did not redo all of the commands and assumed that the resulting psbt was the same?

@AndrewChow I created the same simple transaction in the bitcoin-qt GUI interface to generate the psbt string, I think it should be the same and I tried it several times. The point is that after each restart of Bitcoin Core, the previously imported public key is not inserted into the psbt string.  For example, I imported the public key of address A, and after restarting Bitcoin Core and generating psbt transactions(address A transferring money to address B),  there is no public key of address A in the psbt string.

That's not the same transaction. It will have different inputs, and depending on how you set it up, could be sending coins to somewhere that your wallet isn't watching. Please describe exactly how you are making your transactions - i.e. what buttons you are clicking on and which fields you are filling in. A couple of important things to know are whether you are selecting inputs manually and what you're setting the change address to, if at all. The output of getwalletinfo will also be useful.

@AndrewChow I've updated the issue reproduction action at the bottom of the question, please check it out.


text:
Creating Witness Signature: non-mandatory-script-verify-flag (Signature must be zero for failed CHECK(MULTI)SIG operation)
So I am trying to do my own witness signatures as described in bip143. I have code that correctly verifies the example signature, so I thought I had a hold of the basic signing and verification. I have code that can verify the signatures in bip143 correctly, and using the same code I verify this output correctly, so I am not sure how to determine what is going wrong. The error that I get is
error code: -26
error message:
non-mandatory-script-verify-flag (Non-canonical DER signature)

I suspect that I am not reversing the byte order in something, but I am really not sure how to further diagnose this.

----unsigned transaction-------
020000000102303e285bb228344346acb5292397a761103efcd0182c4cf95b4d01b2bdda400000000000ffffffff02400d03000000000016001453f0b90d7be69d4f5f1356908a9e12fef93561df74e0022a010000001600140f54e16bd59da94840741b69ab9b9db99467a85800000000
-------end unsigned transaction------
dhash(prevouts)=02303e285bb228344346acb5292397a761103efcd0182c4cf95b4d01b2bdda4000000000
dhash(sequences)=ffffffff
dhash(outputs)=400d030000000000001453f0b90d7be69d4f5f1356908a9e12fef93561df74e0022a0100000000140f54e16bd59da94840741b69ab9b9db99467a858
nversion:      02000000
hashPrevouts: 9ebc8589966e2dd13cd64af1835262c2d8e931b4388a8ebe76620814b4f407bd
hashSequence: 3bb13029ce7b1f559ef5e747fcac439f1455a2ec7c5f09b72290795e70665044
outpoint:     02303e285bb228344346acb5292397a761103efcd0182c4cf95b4d01b2bdda4000000000
scriptCode:   1976a914992fda25fdf8447092aa223a34bedd3572173d8688ac
amount:       00f2052a01000000
nSequence:    ffffffff
hashOutputs   a54e6a8744fa773a5f1c64d7f60d53efd69f8d76e85e7817970a87990874e4c1
nLockTime     00000000
nHashType     01000000
pre hash image: 020000009ebc8589966e2dd13cd64af1835262c2d8e931b4388a8ebe76620814b4f407bd3bb13029ce7b1f559ef5e747fcac439f1455a2ec7c5f09b72290795e7066504402303e285bb228344346acb5292397a761103efcd0182c4cf95b4d01b2bdda40000000001976a914992fda25fdf8447092aa223a34bedd3572173d8688ac00f2052a01000000ffffffffa54e6a8744fa773a5f1c64d7f60d53efd69f8d76e85e7817970a87990874e4c10000000001000000
dhashed image: 4f8fce636bb4a44aa6a241e9597bbe54fce3bfe3426a49a94c5b006d42eac007
rawsig: 2cbb748d06a1b87c933a3118d8c2a5a0ab02111bae777a9e52ba8571eee2d549e80a6e4cef24a172e64d11aa7b83c8005fe6fc078307b9e663ef9eb14fce05a3
der sig: 304402202cbb748d06a1b87c933a3118d8c2a5a0ab02111bae777a9e52ba8571eee2d5490220e80a6e4cef24a172e64d11aa7b83c8005fe6fc078307b9e663ef9eb14fce05a3
script pub key: 992fda25fdf8447092aa223a34bedd3572173d86
pub key: 0392ff36d0ae9f3a74c0483fd309ff9144972b1dce6d6dfe4d9de474c721b36521
----signed transaction-------
0200000000010102303e285bb228344346acb5292397a761103efcd0182c4cf95b4d01b2bdda400000000000ffffffff02400d03000000000016001453f0b90d7be69d4f5f1356908a9e12fef93561df74e0022a010000001600140f54e16bd59da94840741b69ab9b9db99467a8580247304402202cbb748d06a1b87c933a3118d8c2a5a0ab02111bae777a9e52ba8571eee2d5490220e80a6e4cef24a172e64d11aa7b83c8005fe6fc078307b9e663ef9eb14fce05a301210392ff36d0ae9f3a74c0483fd309ff9144972b1dce6d6dfe4d9de474c721b3652100000000
-------end signed transaction------

This is running on my regtest. I can provide any additional info that people might think would be helpful. I have spent many hours comparing every bit possible to all the examples before posting here. I know it says non connical der sig, but my researching, other people say that error happens when many different things go wrong. Also when I compare the signature to the core signed ones they are an exact match in format with the "30440220" then 32 bytes then 0220 then the other 32 bytes. The only thing that is different is the actual sig data. Plus I get the same error when using a native connical DER output.  Any help is greatly appreciated as always

-----UPDATE-----
so the problem with the above script is that mainly it is not DER encoded correctly, meaning that if s or r most MSB > 0x80 it treats it as an negative because it is a signed integer. In this example the MSB of S is 0xE8  which is greater then 0x80. bitcoin has a policy that it actually must be less than the ecdsa curve value "n". First below is the code that corrects the problem, and I can verify the signature (but the error still exist in bitcoin core). Code to invert S in python:
n=115792089237316195423570985008687907852837564279074904382605163141518161494337
s = bytearray(s_bytes)
sint = 0
cnt = 31
for bb in s:
    sint += bb << cnt*8
    cnt -= 1
if sint >= n/2:
    print("Negated")
    sint = n-sint
    s = bytearray(sint.to_bytes(32, byteorder='big'))

not to bad. And as for r the bitcoin core just re-signs until it is lower than 0x80. it is legal for it to be higher if the der encoder correctly pads the MSB). So did it the same way. below is a new transaction that has the problem corrected, though I still get the same error :-/.

----unsigned transaction-------
020000000102303e285bb228344346acb5292397a761103efcd0182c4cf95b4d01b2bdda400000000000ffffffff02400d03000000000016001453f0b90d7be69d4f5f1356908a9e12fef93561df74e0022a010000001600140f54e16bd59da94840741b69ab9b9db99467a85800000000
-------end unsigned transaction------
dhash(prevouts)=02303e285bb228344346acb5292397a761103efcd0182c4cf95b4d01b2bdda4000000000
dhash(sequences)=ffffffff
dhash(outputs)=400d030000000000001453f0b90d7be69d4f5f1356908a9e12fef93561df74e0022a0100000000140f54e16bd59da94840741b69ab9b9db99467a858
nversion:      02000000
hashPrevouts: 9ebc8589966e2dd13cd64af1835262c2d8e931b4388a8ebe76620814b4f407bd
hashSequence: 3bb13029ce7b1f559ef5e747fcac439f1455a2ec7c5f09b72290795e70665044
outpoint:     02303e285bb228344346acb5292397a761103efcd0182c4cf95b4d01b2bdda4000000000
scriptCode:   1976a914992fda25fdf8447092aa223a34bedd3572173d8688ac
amount:       00f2052a01000000
nSequence:    ffffffff
hashOutputs   a54e6a8744fa773a5f1c64d7f60d53efd69f8d76e85e7817970a87990874e4c1
nLockTime     00000000
nHashType     01000000
pre hash image: 020000009ebc8589966e2dd13cd64af1835262c2d8e931b4388a8ebe76620814b4f407bd3bb13029ce7b1f559ef5e747fcac439f1455a2ec7c5f09b72290795e7066504402303e285bb228344346acb5292397a761103efcd0182c4cf95b4d01b2bdda40000000001976a914992fda25fdf8447092aa223a34bedd3572173d8688ac00f2052a01000000ffffffffa54e6a8744fa773a5f1c64d7f60d53efd69f8d76e85e7817970a87990874e4c10000000001000000
hashed image: 4f8fce636bb4a44aa6a241e9597bbe54fce3bfe3426a49a94c5b006d42eac007
rawsig: 38d610fa975440d09b501864185e656546015ee98dd8b5b58eb41be547b6a686be784798307a123bdfefce7463b6e7203b69d18950ea95ae684dddb5a9bcaeaf
s before: be784798307a123bdfefce7463b6e7203b69d18950ea95ae684dddb5a9bcaeaf
Negated
s after: 4187b867cf85edc42010318b9c4918de7f450b5d5e5e0a8d578480d726799292
der sig: 3044022038d610fa975440d09b501864185e656546015ee98dd8b5b58eb41be547b6a68602204187b867cf85edc42010318b9c4918de7f450b5d5e5e0a8d578480d726799292
script pub key: 992fda25fdf8447092aa223a34bedd3572173d86
pub key: 0392ff36d0ae9f3a74c0483fd309ff9144972b1dce6d6dfe4d9de474c721b36521
----signed transaction-------
0200000000010102303e285bb228344346acb5292397a761103efcd0182c4cf95b4d01b2bdda400000000000ffffffff02400d03000000000016001453f0b90d7be69d4f5f1356908a9e12fef93561df74e0022a010000001600140f54e16bd59da94840741b69ab9b9db99467a85802473044022038d610fa975440d09b501864185e656546015ee98dd8b5b58eb41be547b6a68602204187b867cf85edc42010318b9c4918de7f450b5d5e5e0a8d578480d72679929201210392ff36d0ae9f3a74c0483fd309ff9144972b1dce6d6dfe4d9de474c721b3652100000000
-------end signed transaction------
Again, help is appreciated

That is not valid DER encoding. When the top (first) byte is >= 0x80, the represented number is interpreted as negative. When the value encoded is larger than 2^255, that means an additional 0 byte in front must be added. The Bitcoin network actually has an additional constraint, which requires the S value in the signature to be below n/2. This is very easy: when s >= n/2, negate it (replace with n-s).

Sorry I had to try it out. ok so when I was using the native canonical DER formatting it did sometimes have the leading 0 and was one longer. so to make sure I understand, basically when it is bigger than 0x80 instead of just adding in the 00 we instead negate the entire S ? so if s[0] > 0x80,  s = 0xFFFFFFFF...-s?

@PieterWuille I tried what I thought was correct. no success yet. negating S when MSB is 1 and not adding the leading zero it won't validate in general now. How does bitcoin core know if you have inverted it or not?

The reason for doing this is malleability: we don't want 3rd parties relaying a transaction to change it such that it txid changes but without invalidating it. It turns out that for ECDSA, if (r,s) is a valid signature, then so is (r,n-s). To avoid that malleation, Bitcoin Core has a policy rule that it only relays transactions with signatures whose s value is < n/2. It does not need to know whether it was negated or not, as both forms are equally valid ECDSA.

Note that n is a rather nontrivial number, it's around 2^256 - 1.5*2^128 (search for secp256k1 curve order), and R and S and encoded in big endian notation.

ok I got s-n and my test validation program can validate the signature still. For bitcoin core, what about r? if its to big for the max value, are we ok, just adding the leading 0 ? and following a "default" der scheme?

Yes, R just has to be in range [0,n). Since there is a 50% chance that the value is below 2^255 however, you can save a byte often by just trying to sign again. That's what Bitcoin Core does, but just for financial reasons (fee scales with signature size); it's perfectly legal to have 33-byte R values.

@PieterWuille ok So I have updated the example with the corrections. it still decodes fine, and I can still validate the signature myself. sendrawtransaction still has the same error :-/

What is your DER encoded signature now?

3044022038d610fa975440d09b501864185e656546015ee98dd8b5b58eb41be547b6a68602204187b867cf85edc42010318b9c4918de7f450b5d5e5e0a8d578480d726799292
I mean its a new signature because of the randomness. I think everything else should be the same. In the update I posted the original and the negated S just for completeness

That looks correctly DER encoded. If you still get the same problem, I have no idea.

@PieterWuille alright well I appreciate the help/check. thank you

ok figured it out! after about a week :-D.  I want to post this because I spent the last week learning the bitcoin core to output the intermittent values during the sighash to compare. Anyone else needing to really understand how the core works this will hopefully be really helpful. So first my specific problem: my output script was just completely incorrect, I think I got the locking and the unlocking script mixed up, which still shouldn't invalidate the signature, but then also you notice in the example the output is like:
202cb20600000000 1976a9148280b37df378db99f66f85c95a783a76ac7a6d5988ac

well the first 0x19 is not part of the script public key (the way decode raw transaction displays it). So in the description where it says

If the sighash type is neither SINGLE nor NONE, hashOutputs is the double SHA256 of the serialization of all output amount (8-byte little endian) with scriptPubKey (serialized as scripts inside CTxOuts);

you hash with the length prefix 0x19. I was including the length in the overall transaction so it still decoded fine, but not in the way I hashed. Simple misunderstanding.
Below is how I built and modified the core (0.21.1) to output the values to figure this out

I mostly followed this example https://jonatack.github.io/articles/how-to-compile-bitcoin-core-and-run-the-tests, basically installed most the dependencies from that list
export BDB_PREFIX='/home/ubuntu/btc/bitcoinsource/bitcoin-0.21.1/db4'
./configure BDB_LIBS="-L${BDB_PREFIX}/lib -ldb_cxx-4.8" BDB_CFLAGS="-I${BDB_PREFIX}/include" --enable-debug
make
Notice the use of --enable-debug. I think this was needed to let you use gdb? I did occasionally get thread lock errors in the log file which would cause it to crash in debug mode when using the wallet api.
I created a separate bin folder with some sym links to the 4 main binary executables. then edited /etc/environment to append the path to that folder. then ran "source /etc/environment". This lets you run the binaries from anywhere.
now you can run bitcoind -regtest -daemon. From here you can go into the source  code where the binaries were built and run gdb, and attach to the process.
Modify <bitcoin_dir>/src/script.interpreter.cpp At the top I wrote some basic function to take a few different variable types and print to hex. Apologies for the bad code, I am C person by trade, not C++. I created a separate logging file so i didn't mess with bitcoind's main log

std::string hexhalftostr(unsigned char val)
{
    if (val <= 9)
        return std::to_string(val);
    else if(val ==10)
        return "A";
    else if(val ==11)
        return "B";
    else if(val ==12)
        return "C";
    else if(val ==13)
        return "D";
    else if(val ==14)
        return "E";
    else if(val ==15)
        return "F";
    else return "";
}
void logbytes(unsigned char *data, int len)
{
    std::ofstream myfile;
    myfile.open ("/home/ubuntu/btc/bitcoinsource/bitcoin-0.21.1/special.log", std::ios_base::app);
    std::string ss = "";
    for (int i = 0;i < len;i++)
    {
        char lsb, msb;
        lsb = 0x0f & data[i];
        msb = 0x0f & (data[i] << 4);
        ss += hexhalftostr(msb) + hexhalftostr(lsb);
    }
    myfile << ss;
    myfile.close();
}
void logbigint(uint256 data)
{
    logbytes(data.data(), 32);
}
void logint(unsigned int data)
{
    unsigned char arrayOfByte[4];
    for (int i = 0; i < 4; i++)
        arrayOfByte[i] = (data << (i * 8));
    logbytes(arrayOfByte, 4);
}
void loglong(unsigned long data)
{
    unsigned char arrayOfByte[8];
    for (int i = 0; i < 8; i++)
        arrayOfByte[i] = (data << (i * 8));
    logbytes(arrayOfByte, 8);
}
void logtext(std::string ss)
{
    std::ofstream myfile;
    myfile.open ("/home/ubuntu/btc/bitcoinsource/bitcoin-0.21.1/special.log", std::ios_base::app);
    myfile << ss;
    myfile.close();
}

SignatureHash is the function where we can actually output them as they are being computed

 logtext("\nnVersion:     ");
        logint(txTo.nVersion);
        logtext("\nhashPrevouts: ");
        logbigint(hashPrevouts);
        logtext("\nhashSquence:  ");
        logbigint(hashSequence);
        logtext("\noutpoint:  ");
        logbigint(txTo.vin[nIn].prevout.hash);
        logint(txTo.vin[nIn].prevout.n);
        //can't figure out how to grab the bytes
        ////logtext("\nscriptCode:  ");
        ////logbigint(scriptCode);
        logtext("\namount:  ");
        loglong(amount);
        logtext("\nnSequence:  ");
        logint(txTo.vin[nIn].nSequence);
        logtext("\nhashOutputs:  ");
        logbigint(hashOutputs);
        logtext("\nnLockTime:  ");
        logint(txTo.nLockTime);
        logtext("\nhashType:  ");
        logint(nHashType);

And the return of the function had to be replaced because the sha256 hash consumes the data so
 return ss.GetHash();

Becomes
 uint256 sighash = ss.GetHash();
 logtext("\nhashed image: ");
 logbigint(sighash);
 logtext("\n");
 //return ss.GetHash();
 return sighash;

I would recomend setting a break point and stepping through "EvalScript" function. This is where all the magic happens, and is super good for educating yourself on how the actual bitcoin script runs, because each error code encompasses virtually unlimited mistakes, I am not sure there is any other way to debug your code without doing this.


text:
The example signature in BIP143 doesn't validate?
In BIP143 the first example
has a witness signature and it doesn't appear to validate/be correct?! I wrote a super simple python program to demonstrate
import ecdsa
import asn1

#Importing the 3 pieces of data from the example to byte arrays
pub = bytearray.fromhex("025476c2e83188368da1ff3e292e7acafcdb3566bb0ad253f62fc70f07aeee6357")
sighash = bytearray.fromhex("c37af31116d1b27caf68aae9e3ac82f1477929014d5b917657d0eb49478cb670")
dersigscript = bytearray.fromhex("304402203609e17b84f6a7d30c80bfa610b5b4542f32a8a0d5447a12fb1366d7f01cc44a0220573a954c4518331561406f90300e8f3358f51928d43c212a8caed02de67eebee")

#deocding the DEC encoding
decoder = asn1.Decoder()
decoder.start(bytes(dersigscript))
tag, sigscript = decoder.read()

#stripping off the script  potion so we just have a signature
sig = bytearray(sigscript)[2:66]

vk = ecdsa.VerifyingKey.from_string(pub,  curve=ecdsa.SECP256k1)
vk.verify(sig, sighash)

If anyone can shed some light on what stupid thing I am doing, or an example reference that breaks it down, so it works, that would be really helpful. I have also tried reversing byte order on all 8 combinations of the 3 inputs. That being said I have used the private and public key posted to do my own signatures/validation, so I am fairly confident they are in correct order, just less sure about the sig.
Thank you!!
----EDIT-----
Just wanted to list other things that were tried for completeness sake

as pieter pointed out I was incorrectly decoding the private key. S and R are split up into 2 different variables

304402203609e17b84f6a7d30c80bfa610b5b4542f32a8a0d5447a12fb1366d7f01cc44a0220573a954c4518331561406f90300e8f3358f51928d43c212a8caed02de67eebee
becomes
3044 02 20 3609e17b84f6a7d30c80bfa610b5b4542f32a8a0d5447a12fb1366d7f01cc44a 02 20 573a954c4518331561406f90300e8f3358f51928d43c212a8caed02de67eebee
so the raw 64 byte sig is 609e17b84f6a7d30c80bfa610b5b4542f32a8a0d5447a12fb1366d7f01cc44a573a954c4518331561406f90300e8f3358f51928d43c212a8caed02de67eebee
unfortunately that still did not work

Additionally I tried using pythons native der style ecdsa which was still not successful
from ecdsa.util import sigencode_der, sigdecode_der
vk.verify(bytes(sig), bytes(sighash), sigdecode=sigdecode_der)
and
vk.verify(bytes(sig), bytes(sighash), hashlib.sha256, sigdecode=sigdecode_der)
where sig starts with 0x30:
3044022047ac8e878352d3ebbde1c94ce3a10d057c24175747116f8288e5d794d12d482f0220217f36a485cae903c713331d877c1f64677e3622ad4010726870540656fe9dcb

what is interesting is the python library didn't give me any errors except bad signature. So I can't seem to find any clues yet as to how to do this with a standard library. If I do figure it out I will post an answer

Using this library:
from cryptotools import Signature, PublicKey, hex_to_bytes

pub = PublicKey.from_hex('025476c2e83188368da1ff3e292e7acafcdb3566bb0ad253f62fc70f07aeee6357')
sig = Signature.from_hex('304402203609e17b84f6a7d30c80bfa610b5b4542f32a8a0d5447a12fb1366d7f01cc44a0220573a954c4518331561406f90300e8f3358f51928d43c212a8caed02de67eebee')
sighash = hex_to_bytes('c37af31116d1b27caf68aae9e3ac82f1477929014d5b917657d0eb49478cb670')

>>> sig.verify_hash(sighash, pub)
True

Wait wait wit. Is the reason why I never see this done with a standard library because the bitcoin signature algorithm is not RFC compliant ?!?!  From here https://en.bitcoin.it/wiki/Elliptic_Curve_Digital_Signature_Algorithm it says the signature is 71-73 bytes for the signature which is not correct. This is mind boggling

@noone392 which RFC are you refering to? Afaik bitcoin signatures follow the DER encoding specification https://github.com/bitcoin/bips/blob/master/bip-0062.mediawiki#der-encoding

it is compatible with DER encoding, I mean the actual ECDSA specification like RFC 6979. the signature must be exactly 64 bytes to be SECP256k1 compliant. And the provided signature, while it matches the  wiki on it, is not the correctly length to be compliant. Basically I don't think you can call this ECDSA rather its a bitcoin signature that uses elliptic curve math.

Bitcoin uses ECDSA, with DER-encoded signatures. It follows those specifications exactly. ECDSA is a signature algorithm, not the encoding.

@PieterWuille what specification are you referring to exactly? because in my example I DER decode it and use what I believe is the raw 64 byte key portion.

Ok, your conversion from DER to 64-byte signature format is incorrect. A DER signature consists of a 2-byte "struct marker" (saying that a tuple follows), then a 2-byte "integer marker", followed by the R integer (variable number of bytes), then another 2-byte "integer marker", followed by the S integer (variable number of bytes). You need to extract the R and S elements, pad them to 32 bytes each (by prepending/removing zeroes up front), and then concatenate them.

You can probably do that with the asn1 module by decoding the "sigscript" variable you get out (not really appropriately named, it's just a tuple of signature elements, no script involved) again with asn1.

@PieterWuille Thank you for finding that issue. you help is very appreciated. For some reason I remember ecdsa sig being a single variable.... anyway I have the 2 32 byte pieces connected together now that I am parsing them. Unfortunately it still doesn't verify. I also tried using python's sigdecode=sigdecode_der to provide the der encoded format but also no luck...

The problem is likely that you need to use VerifyingKey.verify_digest rather than .verify. The latter hashes the messages again, while you're already providing a hashed message.

@PieterWuille That did it :-D I will post the answer. Thanks again

The standard for ECDSA is X9.62, as clearly stated in RFC6979, and does not specify any encoding for the signature. RFC6979 in 2.4 step 4 notes, correctly, it is "common" to use DER-encoded ASN.1, but that's not a requirement, and it absolutely isn't fixed-length. The sample code in A.3 does DER, but that's only a sample. Bitcoin uses DER for transaction signatures, but the fixed-length 'plain' format for message signatures; see https://bitcoin.stackexchange.com/questions/12554/ and https://bitcoin.stackexchange.com/questions/90799/

@dave_thompson_085 Thank you for the clarification

ok So thanks to PieterWuille I (we) were able to figure out why my code wasn't working and how to use a standard python or openssl library.
These were the following problems

the DER decoding for the python library was used incorrectly. We needed to iterate through all the variables and pull the two pieces of the signature (r and s) our separately and recombine them.
Because the standard signing libraries have the hash built into the sign/verify, and bitcoin uses a double hash, to make it work with a standard library, you need to hash 1 time manually then provide it to the library to perform the second hash. This means we could not use sigHash as an input, but rather has to use the hash preImage.
Posted below is about the shortest python script possible showing how to verify the signatures with the examples given in both raw format and DER encoded format:

import ecdsa
import asn1
import hashlib
#used for der version
from ecdsa.util import sigencode_der, sigdecode_der

#Importing the 3 pieces of data from the example to byte arrays
pub = bytearray.fromhex("025476c2e83188368da1ff3e292e7acafcdb3566bb0ad253f62fc70f07aeee6357")
DerEncodedSig = bytearray.fromhex("304402203609e17b84f6a7d30c80bfa610b5b4542f32a8a0d5447a12fb1366d7f01cc44a0220573a954c4518331561406f90300e8f3358f51928d43c212a8caed02de67eebee")
prehashimage = bytearray.fromhex("0100000096b827c8483d4e9b96712b6713a7b68d6e8003a781feba36c31143470b4efd3752b0a642eea2fb7ae638c36f6252b6750293dbe574a806984b8e4d8548339a3bef51e1b804cc89d182d279655c3aa89e815b1b309fe287d9b2b55d57b90ec68a010000001976a9141d0f172a0ecb48aee1be1f2687d2963ae33f71a188ac0046c32300000000ffffffff863ef3e1a92afbfdb97f31ad0fc7683ee943e9abcf2501590ff8f6551f47e5e51100000001000000")

#deocding the DEC encoding
decoder = asn1.Decoder()
decoder.start(bytes(DerEncodedSig))
tag, sigder = decoder.read()

sig1 = sigder[2:34]
sig2 = sigder[36:68]
sig = sig1+sig2

#Now we need to get the single sha256 hash of the preimage
m = hashlib.sha256()
m.update(prehashimage)
singlesighash = m.digest()

#we won't use the double hash for our verify but this shows you it is the same as sigHash in bip143
n = hashlib.sha256()
n.update(singlesighash)
sigHash = n.digest

vk = ecdsa.VerifyingKey.from_string(bytes(pub),  curve=ecdsa.SECP256k1)
#we have to provide the hash lib because the default is not sha256
result = vk.verify(bytes(sig), bytes(singlesighash), hashlib.sha256)
print(result)
#this is the der native way and does not require us to der decode the signature but is compatible with openssl
result = vk.verify(bytes(DerEncodedSig), bytes(singlesighash), hashlib.sha256, sigdecode=sigdecode_der)
print(result)


text:
If bitcoin value is going down do you lose anything or only when you sell like the stock market?
If Bitcoin is going down, do I only lose if I sell my bitcoin?
So, if I hold them I won’t lose?

Well basically, think of it as if u had an apple. If you decide at any moment that you want to sell it, you get a specific price for it, which is the current price of an apple.
So if you bought 1 bitcoin when 1 bitcoin was worth 18K USD and sold it at 27K USD you have now a profit of 50% which means that you now have 27K USD as fiat in your hand instead of having 18K (which you had when you bought a single BTC worth 18K back then)
This works the same if you buy it at 27k and sell it at 18k, except, you then lose 33% of your money that you invested in it when you bought it. That means that if you buy 1 BTC at 27K and sell it at 18K because maybe you are scared to lose more, you end up losing 9K because you sold it.
If you don't sell it, you will remain with that 1 BTC the whole time when it's price fluctuates, but it is basically worth more (or less) depending on it's current market price.

Thankyou Pinkovai for your in depth explanation I just wanted to be sure if I hold and bitcoins going down my holdings will be safe til it goes back up and I can decide to sell for profit if that makes sense

So basically it is same as stock market

@chytrik thanks for the correction, my bad of that math :))

@Sandy if you are satisfied with my answer, you can mark it as the answer for this question. And yes, same as stock market in that way.

Yes, you only really take a loss (or make a gain) in terms of dollars or other fiat currencies when you trade whatever cryptocurrency you're holding back in for fiat.
And yes, it's similar to other assets that can be traded for fiat currency, like stocks, bonds, other fiat currencies, houses, apples, etc. "On paper," as they say, like if you were keeping accounting records to estimate the net worth of all your investments, the current value of that investment has gone down and so the net worth of your investments goes down. But you haven't "realized" the loss until you've sold the asset at its current, lower price. So if you can hold it until the price goes back up again, you can "realize" a gain instead of a loss.


text:
Proof of private keys ownership for multiple addresses
In a two party coinjoin consists of Mr Seller and Mr Buyer.
Mr Seller has set A of sender addresses/input and set B of receiver addresses (output)
Mr Buyer has set C of sender addresses/input and set D of receiver addresses (Output)
Assume all input/output are addresses (P2PK or P2PKH and none of them are script).
How does Mr Seller prove to others that set A and set B addresses belong to him? Is there any other way to prove than proving the address one by one which can be very long. On top of that set B addresses are hashes.
Mr Seller also need to cooperation from Mr Buyer to provide proof that
set C and Set D address belong to Mr Buyer.
This is a requirement from IRS audit department who wanted proof of payment amount from
the given invoice record.

I think what you need is a threshold signature(n of n).
I don't know of any tools that would help you create such signature, but conceptually speaking it is not that difficult.
You take all of the addresses in Set A and Set B and get the corresponding public keys. Then you can create a single threshold signature(MuSig for example) using all of those public keys.
Since you are asking these for tax purposes, you would also need to prove that you don't own those coins in Set(C+D). For that, you would also need another threshold  signature from other person over Sets(C + D) to show that they own those coins.
As I said earlier, I don't know of any readymade tools for doing these.

Why would you prove ownership of inputs/outputs in two-party coinjoin? This data upon publishing would make the coinjoin efforts obsolete and would totally cancel all anonymity set in this specific transaction.
For input you could use some specific wallet setup where inputs would be in separate xpub derivation path, and outputs in separate.
Even with point 2. you'd have to post the outputs as the list of addresses if you don't want to leak pubkeys?

This looks to me like an XY problem and your requirements are contradicting themselves. Could you explain what you're trying to achieve?

for auditing and tax purposes! IRS asked for the proof of payment details. Let assume IRS will not leak out the info.


text:
How miner collects their transaction fee in Confidential Transaction
These are followup questions from
Transaction Fee in Confidential Transactions
In Confidential Transaction, Alice own 10btc sends 1 btc to Bob
Input : C1 = Commit (10btc)  = 10 G + 10 H
Output: C2 = Commit (1 btc)  = 1 G + 1 H
C3 = Commit (8.999 btc) = 8.999 G + 8.999H
C4 = Commit (stated as fee) (0.001 btc) = 0.001G + 0.001H
C1 = C2 + C3 + C4. (for simplicity reason, the r (blinding value rG) is the same as v (actual btc value vH).
My questions are
Q1. Is the winning miner is pre-determined already?
Q2. If Q1 = Yes? Then what is the coinbase address?
Q3. If Q1 = No? Then how does Alice know which address to send the fee to?
Q4. Suppose the fee is not yet sent to the winning miner yet? Then how does the winning miner know the blinding value of r = 0.001 since Alice don't know which miner to tell. Or the r and v values are not encrypted at all for this case.

Q1. That depends on the blockchain's consensus rules. It has nothing to do with Confidential Transactions. In Blockstream Liquid, the miners (really, federation signers) rotate, and one by one propose a block which 2/3s of the other signers need to agree on. There are other systems that use some variant of Confidential Transactions that work completely differently.
Q2. Whatever the miner wants.
Q3. Alice doesn't send the fee to any address. Her transaction simply lists inputs, outputs, and fee (the fee is not an output, it has no address; it's just a field that says "the fee is 0.001", in cleartext - no Pedersen commitment). In your example, the transaction would list {inputs: [utxo with output C1,witness], outputs:[(C2,address1),(C3,address2)], fee:0.001}, and the validity rule would be C1 = C2 + C3 + 0.001*H).
Q4. There is no blinding for the fee, so there is nothing the miner needs to know.

Thank You very much Pieter. Now, it very super clear.


text:
Why distinguish between receiving address and change address
I see receiving address and change address in wallet client and bip44. Why distinguish between receiving address and change address?
What will be the problem if I use only change address chain for receiving and changing, or use only receiving address chain for receiving and changing?

BIP-32 says

In the example of a webshop, the webserver does not need access to all public keys of the merchant's wallet; only to those addresses which are used to receive customer's payments, and not for example the change addresses that are generated when the merchant spends money. Hierarchical deterministic wallets allow such selective sharing by supporting multiple keypair chains, derived from a single root.

The network doesn't care. It doesn't distinguish the two. Any address (i.e. locking script) can be used or reused for either purpose.
Your wallet might care though.

This distinction only exists for the internal processes of your wallet, and to provide a better UX to the user.
Consider for example when you request a payment. You generate a receive address, the sender creates a transaction to pay you. The transaction creates a recipient output that assigns the payment to your receive address. Your wallet will now show this payment in your “Receive” tab.
On the other hand, when you make a payment, your transaction might have two outputs. One recipient output to make the payment, and a change output to return the remainder of the funds from the inputs to your own wallet. The change address to receive the change output was created on the fly by your wallet when it created the payment transaction. The change address and change output will not show up in your “Receive” tab, because you transferred funds from your left pocket to your right pocket. Rather the transaction will show up in your “Send” tab.
Your wallet might also use a different approach when scanning receive and change addresses. While you might give out a receive address, but never receive a payment to it, almost every change address on the subchain will be used, since they are only generated on the fly when you are already making a payment.
So, while they work exactly the same under the hood and there is no mechanical issue using one or the other for the opposite purpose, in regard to actually receiving the funds, your wallet software may operate under some assumptions regarding the purpose of the addresses that could impact your UX. I would recommend using the addresses per the purpose that they are meant to have.


text:
Total Decentralization
I am fairly new to Bitcoin and do not know the ins and out of using the Electrum wallet(i was recommended this wallet by bitcoin.org). I see on the internet that there are several methods by which I can send BTC from person 1 to person 2 like liquid network and there's something called BECH32 and SEGWIT. I do not know about all this but I want a suggestion as to how do I maintain complete decentralization of the funds in my wallet AND all the transactions that I am making from my wallet. I do not want any third party to be able to even remotely control my funds anyhow whatsoever. Decentralization is top priority for me. I want to use the bitcoin chain the old school way
Thanks

Welcome Anuj,
My recommendation would be to set up and run your own bitcoin full node and connect your Electrum wallet to it. This will provide you with complete autonomy and allow you to verify all transactions independently. By running your own node, you can ensure that you have full control over your transactions and can avoid any potential security risks associated with relying on third-party nodes.
While having complete control over your funds can be advantageous, it also comes with the risk of losing all your funds if you are not careful. Therefore, it is essential to have a thorough understanding of the system before storing a significant amount of value in it.
Regarding the terms SegWit vs Native SegWit (Bech32) - these are two address formats used in bitcoin transactions. There are other address formats in use as well.
You may find it helpful to visit Jameson Lopp's website, which provides a curated list of bitcoin resources: https://www.lopp.net/bitcoin-information.html -- This site offers a wealth of information on full node setup and other aspects of the bitcoin ecosystem.
If you have any questions or concerns, do not hesitate to reach out. The folks here are always willing to assist and provide guidance.

Thanks for your reply but I am new to Bitcoin. Can you give me a way to hold and transact with BTC without having to run my own set up and invite some element of risk? I just want to hold BTC in my wallet. Also which address format is better to use? there are ways and means to conduct faster and cheaper transactions on BTC chain which also means centralization which I want to avoid. can you tell me what should I avoid doing during transacting BTC?

If your goal is to simply hold ₿ in your wallet, it may be worth exploring non-custodial wallet options. Wallets can generally be divided into two categories: custodial and non-custodial. Custodial wallets involve a third party having control over your private keys, while non-custodial wallets give the user complete control over their assets. While custodial wallets may offer convenience and ease of use, non-custodial wallets provide greater security and control over your funds. The choice between these two options often comes down to a tradeoff between convenience and responsibility.

You can also use the Electrum wallet, and connect it to a third-party node. Alternatively, for beginners, mobile wallets like BlueWallet or Wallet of Satoshi may be a good starting point.
Regarding SegWit and Bech32 addresses - Bech32 has a comparatively lower transaction fee. Following post provides more details, but it may be too technical for you: https://bitcoin.stackexchange.com/q/100434/87158
Start with a small amount of funds that you are willing to lose and gradually build up to a more secure setup as you become more familiar with the technology.

Why does Electrum wallet have the option for multiple addresses? Why cannot I have 1 address assigned to my wallet by default? Also there are expiring wallet addresses from 10 minutes to 1 week and also there's the option of 'never' which should be there by default. is there a way to have 1 permanent wallet address?

Using a single receiving address can compromise your privacy, as it allows others to track your transactions and activity on the blockchain. It is recommended that you generate a new receiving address for each transaction. This practice enhances the security of your funds by making it more difficult for others to track your payments and transactions.


text:
Is it safe to reuse my seed to create a different wallet that supports Bitcoin?
I created a wallet for Ether using MyEtherWallet. I want to store Bitcoin now, but MyEtherWallet does not support it.
Since I prefer not having two remember two different seed-phrases, I was thinking of just reusing the 12 word seed-phrase of MyEtherWallet to create a new wallet on one that does support Bitcoin, for example Electrum (I know that Electrum doesn't use the standard BI39 protocol).
I can't think of any problem with doing this, but I'm kind of new to this, so maybe there's something I haven't thought of.
Is there something I'm missing here? Is doing this considered unsafe?
EDIT: I must add that for the time being and for the foreseable future, I do not plan to spend any crypto, only accumulate. My seed-phrase was generated on an old laptop that will never be connected to the internet. The only thing that ever sees the light of day is my public address, which I use to send crypto to my wallet.

That would be a bad idea for multiple reasons:

It increase the risk of exposing your seed phrase since you gonna have to use it more
It limit recovery option, if you lost your seed phrase, you will loose access to both wallet
if one of your wallet software has a security vulnerability, you will lost fund on both chain

Please see the edit.


text:
Get balance and TXs
How to get balances and transactions of a Bitcoin address? - Devsuite

Does this answer your question? How to get an address's balance with the bitcoin client?

to see the balance from your Bitcoin address, you can use QuickNode.
Here is an example:
https://www.quicknode.com/guides/quicknode-products/apis/look-up-the-address-balance-for-a-wallet-for-bitcoin-on-quicknode
Irwing Tello


text:
How are bitcoin transactions and the blockchain transferred over the internet?
So I have several specific question connected to this topic and I'm aware of the conceptual answers and I'm only interested in the technical details (with programming terms and concepts as I'm a developer).
My questions are:

If I'm sending a transaction from my wallet application (e.g. electrum), what does it do under the hood? Does it use the HTTP protocol or something else? Or not even use TCP but UDP instead?
How does the miners "listen" to transactions? Do they listening on some port? Is there any generic protocols for "listening" like this?
When a miner mines a block how does he/she broadcast it to the network? Is it the same as sending a transaction from my electrum wallet?

The Bitcoin P2P network
The Bitcoin P2P network is a randomly-wired gossip network. This means that all nodes make arbitrary connections to other peers (using various ways to discover new addresses) using a custom TCP protocol, usually using port 8333. Typical nodes create 8 outgoing connections, and if publicly reachable, accept up to a few 100 incoming connections.
It being a gossip network means that nodes send messages to each other of the form "do you have block with hash X" or "do you have transaction with hash X", after which the peer can respond by asking for the full transaction or block. Simplified, this means that eventually every node learns about every block. The protocol is bidirectional - it doesn't matter who connected to whom; both peers will announce if they have something they don't know the other side already has as well.
Not all network nodes are reachable. Some only make outgoing connections. Sometimes setups are used where someone runs a publicly reachable node, and then runs a second private node that only connects out to their own public node, thereby giving some extra privacy and reduced attack surface.
Not all network nodes validate the blocks and transactions they see and pass on. Nodes that do are called full nodes.
Miners (or at least mining pools) run nodes like this, picking up transactions their nodes learn about, assembling them into candidate blocks, hashing to find a nonce, and if they succeed, broadcast the resulting block out to their peers using the same P2P network. Miners in this way are not special - people don't directly submit things to miners.
Since recently, an extension to the protocol is used to relay blocks much more efficiently, called Compact Blocks. It makes use of the fact that most nodes will already have most of the transactions that are added to a new block.
Specialized relay networks
Additional networks exist of groups of nodes that use specialized protocols to very quickly interconnect nodes. One such protocol is FIBRE, which is in fact UDP based and makes heavy use of error correcting codes to avoid roundtrips.
Electrum
Ironically, Electrum in your question is a bad example. Electrum is not a P2P client (not even a light one). Instead, it uses a client-server architecture, and connects to specialized servers (which are themselves Bitcoin P2P nodes however).

Thanks for the detailed explanation! What about the online block explorers? Can it be done on client side?

I don't understand the question.

Oh, nevermind. Block explorers do everything on the server side.

I see, thanks! Could you please include it in your answer because I addressed it in the question and someone may look for the info in your answer?

Perhaps it's better to move that to a different question; usually you shouldn't ask multiple questions in one thread (as it hurts searchability of answers). I think I could answer the other ones nicely together, but blockexplorers seem pretty unrelated.

Ok, then I'll remove it from the question. I included it because I thought there was a way to communicate with the blockchain from a webpage directly (which turned out the be not the case) and so it fitted with other questions.

So my full node makes the 8 outgoing connections but I've never seen more than ~20 incoming connections.  Certainly not a few 100.  Do I stink?  Or maybe it's because I'm on an rather slow connection - 75 down/15 up - and the other nodes detect that?  (The machine itself isn't slow or under load so I don't think the machine would be contributing to latency ... just the network connection itself?)

(Or maybe it's a config setting I forgot about ...)

@davidbak I said "allow up to". If every node makes 8 outgoing connections, on average each will only have 8 incoming connections too. Many nodes only make outgoing though, pushing the number of incoming connections somewhat higher on those that do support them. But in general, no, 20 incoming connections is very healthy. It's only nodes that are fast and reliable and up for very long periods of time (months or more) that gather 100s of connections. Note that more connections is not in any way better for you - once above a certain number, they just slow you down.

I don't know the answers to all questions but let me try to help (most of the above is not verified).

https://bitcoin.org/en/full-node#port-forwarding
Bitcoin nodes use ports 8333 to listen to new transactions and blocks.
Unfortunately in the picture both TCP and UDP port forwarding is selected so I cannot reason which one is used. I assume it is not HTTP as port 8333 is used. EDIT: As per Pieter's comment TCP is used.
(If you have full node with wallet you do the above. If you have lightweight wallet you send your transaction to some full node that does the above process.)
EDIT per Pieter's comment. The miners (full nodes) listen on port 8333 for inbound connections. Other nodes can establish connection with that node on that port. Also the node itself can establish outbound connections to other nodes that listen for inbound connections on port 8333. It is not necessary to allow for inbound conections. When connection (oubound or inbound) is established between two nodes they exchange transactions and blocks when they a) make transaction, b) mine a block, or c) receive transaction or block from some other node.  Each full node holds several connections. What is the specific protocol, and if some common exist I don't know.
These websites are running servers that have full bitcoin node (more likely more they have more than one node). This node contains all information about the blockchain and pool of pending transactions (as every full node does). The servers crawl this information and store the data and statistics in their databases. The website is just the frontend that shows this information from these databases.
EDIT per Pieter's comment. The node (miner) broadcasts the new block to all nodes it has connection with. More commonly the miner mines in pools so it sends the new block to the pool server, which acts as a node and does the above.

Ok, so if I understand this correctly electrum is a light node and only full nodes can broadcast transaction to the network so it needs an intermediary full node. Otherwise your answer is really insightful and I'm only leaving it open to have some more potential visitors who may can expand your answer with the protocol specifics (tcp, ud, http, etc) but I'll mark this as answer later today.

Electrum is not a node at all. UDP is not used. The protocol is bidirectional, so who connects to whom doesn't matter. Things are not just announced to outbound connections, but to all connections.

Bitcoin works inline with the standard internet protocols we have. When you make a transaction from your wallet application, your wallet provider which hosts a bitcoin core application will submit the transaction on your behalf and add it to its version of the blockchain. Other miners will then begin to verify(mine) that transaction to make sure that the transaction is legitimate. After the Proof of Work is completed by a particular miner, the mined block is added to the longest most authoritative blockchain. This is then propagated across the network so that all the bitcoin core applications can keep themselves updated with the blockchain. The bitcoin core comes with a JSON RPC which can be used to interact with the blockchain to explore the blocks of transactions. You can look at the API documentation
here

As I stated in my question I'm very well aware of the conceptual working of bitcoin and only curious about the technical details which your answer almost entirely lacks. The only (seemingly) useful bit of information is JSON RPC but you didn't provide any details or info on it.

Here is a link to the API docs: https://en.bitcoin.it/wiki/API_reference_(JSON-RPC)


text:
At which stage is a transaction validated?
Let's say Alice owns 12 BTC in her wallet, and she tries to send 5 BTC to Bob. First the transaction is broadcast to all nodes, then transaction is in the transaction pool. Then miners would choose Alice's transaction and add it to their block and find the nonce, and once they've found the nonce, they will broadcast the block with Alice's transaction. In that flow, at which stage is the transaction validated?

Whether while each node receives the transaction ?. Is it the stage , requires 51 % confirmation to validate the transaction?
Otherwise at the stage , once miner found the nonce, he has to broadcast the transaction , other miners will verify the correct combination of nonce.In this stage , does miners need to verify the nonce along with the all the transactions in the block??

@James C.'s answer is correct, but I'll add a bit to address your two questions specifically:

Whether while each node receives the transaction ?. Is it the stage , requires 51 % confirmation to validate the transaction?

No, each node acts independently, and there is no way to measure what percentage of them have accepted a transaction as valid. Because of this, whether or not a transaction is valid has no bearing on its permanence or place in the blockchain record, it is simply a yes/no check on whether the transaction is valid according to the network's rules. Invalid transactions will of course be rejected by the nodes, and thus not relayed through the network.
Any valid transaction can then be included in a new block. Once that happens, the transaction is said to be 'confirmed', though it is prudent to wait for a few confirmations before considering a transaction to be finalized.
Notice the difference between 'validation' and 'confirmation'. It is possible for a valid transaction to never be confirmed in the blockchain (perhaps it has zero tx fees, or somehow is never relayed to a miner), but it is not possible for an invalid transaction to be confirmed.

Otherwise at the stage , once miner found the nonce, he has to broadcast the transaction , other miners will verify the correct combination of nonce.In this stage , does miners need to verify the nonce along with the all the transactions in the block??

A miner will construct a block from valid transactions, and if a valid block is found, it will be relayed to the network. All other nodes and miners will check this block to ensure they have validated each of its parts. Similarly to transaction validation, invalid blocks will be rejected by the network's nodes.

The transaction is validated before its acceptance into the mempool/unconfirmed transaction pool of a node. Newly accepted tx’s are propagated to all connected peers who have not seen it yet.
When a miner begins to mine the next block, it constructs the blocktemplate from the mempool of its full-node. These tx’s are already validated.
Once the newly mined block is found and propagated, it must be validated, along with its included tx’s. If a full-node has already seen and validated a given tx in this block when it was accepted to the mempool, it doesn’t need to repeat this.


text:
Building the merkle root from transactions in a template block
I want to compute the merkle root for transactions in a given template block -  bitcoind:getblocktemplate().
I understand the merkle root is a hash of all the transactions in a block from the last up to the coinbase transaction, but this is currently way above my pay grade and the documentation I can find for this process is abstract and sketchy at best.
Has anyone done this in JavaScript or know of a resource that could be helpful?
Thanks in advance.
client.getBlockTemplate( function(error, template) {
    if (error) return console.log(error);

    // SETUP MINING
    nonce=0;
    const block = {
        version: template.version
        ,previousblockhash: template.previousblockhash
        ,merkleroot: calculateMerkleRoot(template)
        ,time: template.mintime
        ,bits: template.bits
    }

    // START MINER
    miner(block,nonce);
});

It's named after Ralph Merkle, not Angela Merkel.

Oh HAHAHAHAHAHA, fixed.

I'm much closer now I hope this helps someone else trying to do this!
REQUIREMENTS:
NPM bitcoin bitcoin-js merkle-lib

CODE:
client.getBlockTemplate( function(error, template) {
    if (error) return console.log(error);
    txhashes = template.transactions.map(function(tx){ return Buffer.from(tx.txid, 'hex').reverse() });
    merkleRoot = fastMerkleRoot(txhashes, bcrypto.hash256);
    console.log(merkleRoot);
)};

The result is in a Buffer and I'm having trouble converting it to hex.


text:
Beginner Setup Bitcoin Full Node without GUI
I am following the installation steps to setup a bitcoin node, at the steps "Download the blockchain" that required Bitcoin Core GUI which is not able to run on the testing Ubuntu Server without GUI.
The guideline I am following is https://bitcoin.org/en/full-node#ubuntu-1610. At the Ubuntu Installation, under Bitcoin Core GUI.

Is there any command to trigger the download without GUI? Thank you very much.

Welcome to Bitcoin.SE! It would be useful to potential answerers if you could link to the setup guide that you are using in your question. Ordinarily, the blockchain download is automatic, unless something in your setup has changed that.

Thank you so much Willtech, question edited with the guideline I am following. I tried to start the daemon and hope it will download the data but it seems nothing change on H.D. usage so I am guessing it is not downloading. Am I correct?

In the very same guide you linked, follow the instructions for the Bitcoin daemon.

I tried to start the daemon and hope it will download the data but it seems nothing change on H.D. usage so I am guessing it is not downloading. Am I correct?

No, you are not correct. You should check the status of bitcoind itself by using bitcoin-cli to send commands to it. You can use getblockchaininfo to know whether it is actually syncing by calling it twice. If it is syncing, then it will report a different block height and best block hash on the second call.

it works! It is showing blockchain info. However, how can I know it is completed download the full set of data?

RTFM :-) "bitcoin-cli getblockcount" is your first candidate. See possible commands and explanation here: https://en.bitcoin.it/wiki/Original_Bitcoin_client/API_calls_list and here is the same question on bitcoin.SE: https://bitcoin.stackexchange.com/questions/37815/how-to-know-if-bitcoind-synced?noredirect=1&lq=1


text:
Raspiblitz slow sync performance
I'm syncing my raspberry pi 4 model B, 4GB and after 4 days I'm less than 50% synced. It seems as though there is something wrong with bitcoind as there are some weird symptoms going on. The pi is running raspiblitz 1.7RC2 (64bit). I have 400mbps internet and it's connected to a new 1TB sandisk ssd.
First I looked at the debug.log file (sudo tail -f /mnt/hdd/bitcoin/debug.log) When bitoind is first started up I see blocks being added at a rate of several per second. After letting it run for awhile it slows down to one block every few seconds. I also get ping timouts and my peer connections are constantly cutting off and don't exceed ~10.
Trying to communicate with bitcoind also becomes slow, bitcoin-cli getnetworkinfo | grep connections at the start is instant but after awhile this command can take 30 seconds to execute.
Finally using nmon the system usage is very low except bitcoind is reading from the ssd at a rate of ~250 mbps but not doing much writing. The cpu usage is blocked by waiting io. The only thing I have tried was lowering the dbcache by 500mb thinking there wasn't enough free memory but it did not help.
bitcoin.conf:
# bitcoind configuration
# mainnet/testnet
testnet=0
# Bitcoind options
server=1
daemon=1
txindex=0
disablewallet=1
peerbloomfilters=1
# Connection settings
rpcuser=raspibolt
rpcpassword=****
rpcport=8332
rpcallowip=127.0.0.1
rpcbind=127.0.0.1:8332
zmqpubrawblock=tcp://127.0.0.1:28332
zmqpubrawtx=tcp://127.0.0.1:28333
# Raspberry Pi optimizations
dbcache=2560
maxorphantx=10
maxmempool=300
maxconnections=40
maxuploadtarget=5000
datadir=/mnt/hdd/bitcoin

Edit
After several restarts the sync seems to be going faster than before, even after waiting. The disk reads are much lower with some writes as well. Not sure why lower read speeds results in faster syncing...

Eidt 2
I noticed after "leaving block file" it's operating at about 1/2 speed for awhile now. Look at the time stamps in the attached screenshot. Things slow down quite a bit after that.

Less than 50% sync'd after only 4d? So >40%? I'd say that's pretty good. It took me about 4 weeks to sync, though i desired and configured txindex=1. In my experience the early blocks always went quickly, and later blocks slowly (full blocks perhaps?). Also you did not say if you are sync'ing over Tor. I did most of the initial sync (IBD) without Tor because I was having unreliable peering; possibly caused by tor network congestion or outages

I'm not using tor. I lowered my dbindex yesterday and kept monitoring it. went from 50 to 70 last night. I'm pretty convinced there's something getting 'stuck' or a memory leak, etc. Woke up this morning and it's slow again.

Increasing dbcache should increase sync speeds. On a more basic level, it is maybe worth confirming the SSD is plugged into a USB3 port, and not a USB2 port. fwiw, I recently synced btcpayserver onto an rpi4 using an HDD, and it took about 2.5 days to complete.


text:
Coinbase transaction data
Can a coinbase transaction contain only data: 03 - length and 030303 - coinbase itself, or should the coinbase itself contain the height of the block?

https://github.com/bitcoin/bips/blob/master/bip-0034.mediawiki

Add height as the first item in the coinbase transaction's scriptSig, and increase block version to 2. The format of the height is "serialized CScript" -- first byte is number of bytes in the number (will be 0x03 on main net for the next 150 or so years with 223-1 blocks), following bytes are little-endian representation of the number (including a sign bit). Height is the height of the mined block in the block chain, where the genesis block is height zero (0).

in this case, what should the part of the coinbase transaction look like now? example block 511824: ffffffff INDEX
55 LENGTH_COINBASE
0350cf0741d6a6ae79d8f62841d6a6ae79d8dba22f4254432e544f502ffabe6d6da64c1ef0a6cd6ffdecd9966246e336b366c8ba6ef07e9122485bb61b992bf61a8000000000000000fa00b68d1b3d000000000000 COINBASE (scriptsig)
ffffffff SEQUENCE,
where height block?

O, yes, i understund: 03 - ??? what is???
50cf07(hex little endian) == 511824(dec)

okay, i'm little stupid)  03 - length height block and 50cf07 is height block, after 0350cf07 next may come the any text in the range of 100 bytes? what is the encoding that will be deciphered on the sites as a coinbase? I want to write there vbujym


text:
How to generate utxo information for a bitcoin transaction?
I'm using bitcore atm and am trying to set up transactions. but I can't find much information on the txId & the script. I know it says the txId is generated from the transaction, but how can you generate a txId for a transaction which has yet to take place?
Maybe I'm not fully understanding this process fully, is there somebody who understands who could explain it a bit better?
var utxo = {
  "txId" : "115e8f72f39fad874cfab0deed11a80f24f967a84079fb56ddf53ea02e308986",
  "outputIndex" : 0,
  "address" : "17XBj6iFEsf8kzDMGQk5ghZipxX49VXuaV",
  "script" : "76a91447862fe165e6121af80d5dde1ecb478ed170565b88ac",
  "satoshis" : 50000
};

var transaction = new bitcore.Transaction()
  .from(utxo)
  .to('1Gokm82v6DmtwKEB8AiVhm82hyFSsEvBDK', 15000)
  .sign(privateKey);

Edit: I posted this with the misunderstanding of how value is counted on the blockchain. I see now that the txid and/or utxos are not of the current transaction but of the previous unspent transactions that the spending address received.

A transaction ID is just the hash of the transaction, it doesn't matter if the transaction is incomplete (invalid), or unconfirmed, or confirmed. The wTXID is similar in operation, but only hashes a portion of the transaction.

I think OP is confused by thinking the UTXO's txid is that of the spending transaction perhaps.

@PieterWuille You are correct Pieter, I just did an overview of transactions and spending and I didn't fully understand it (still don't but I now understand utxo (unspent outputs). Thank ya'll so much for being patient and not bashing my noob questions. Really do appreciate it.


text:
How is bitcoin (blockchain) secure?
I see a few different levels here and I want to clarify
There's various levels where your password could be potentially exposed... at least from my mind

Storage (paper wallet, metal wallet, cold storage wallet)
Device (Phone, Computer etc.) i.e. Apple, Microsoft, Google (device makers)
Interface (Mobile App, Desktop App) (how do we know which are safe?)
Internet provider? (if it's https secure do we need to worry here?)
Miner? (is it at all possible for miners to manipulate your transaction?)

On what levels are we the most vulnerable?

The question is a little confusing, but:
1 and 2 are more to do with your own personal security practices. Of course, your own bitcoins are only as safe as the measures you take to secure them.
#3 - It is recommended to stick to wallet software that is open source, such that its code can be audited and verified to be safe. If you can read code, you are encouraged to read through the code of the wallet software you use and compile it yourself.
#4 - Traffic on the Bitcoin network isn't encrypted at all. If you run a Bitcoin node, your ISP will be able to see that you're running a Bitcoin node, and if you relay transactions from your node, your ISP will be able to assume those are your transactions. However, they won't be able to modify them, as any modification would require a new signature with your private key.
#5 - At worst, a miner launching a 51% attack can prevent your transactions from confirming. While harmful, they won't be able to actually manipulate your transaction, as your transaction still must have a valid signature from your private key.


text:
How to check coins & confirmations amount programmtically without third party site
How do I check the amount of coins are in a crypto wallet? And how do I check the confirmations?
Is this process different for each cryptocurrency?
I'd like to check these without having to rely on a third party service.
Edit: Am using a nodejs api

You'd use wallet software to manage these things. The procedures depend on what software you're using. There are some recommendations here: https://bitcoin.org/en/choose-your-wallet

@PieterWuille thanks for the info. so I'd need to use a wallet library or something for this?

How do I check the amount of coins are in a crypto wallet?

Bitcoin is not really stored in wallets. The only important data in a Bitcoin wallet is a large random-looking number called a private key that is kept secret and which is used to provide mathematical proof of the right to spend some amount of money.
Amounts are kept track of in a public journal of transactions called the blockchain. Everyone makes their own copy of this transaction journal by exchanging information from it and verifying exchanged information themselves. Verification removes the need for trust.
Some wallet software doesn't keep its own copy of the full journal of transactions - These "lightweight" wallets use Simplified Payment Verification (SPV) and have to trust other wallets to some extent.
Some people don't want to look after their own money and give all their money to some business they trust completely. These businesses own the money on behalf of their customers and do all the keeping track of money for them. Many such businesses fail and their customers then don't get their money back. These types of account are sometimes called "custodial wallets" but they are really accounts with a business and not the Bitcoin wallets envisaged by the inventor of Bitcoin. The inventor of Bitcoin wanted to do away with such businesses.
To work out the amount of money controlled by the private keys in your wallet, you read the transaction journal from the beginning to the end. This allows you to keep track of all Bitcoin money without needing to trust anyone. To avoid reading the whole transaction journal again and again, you make and maintain a separate list of unspent transaction outputs (UTXOs) that you can think of as coins.
To count the number of confirmations of a transaction you count the number of blocks in the transaction journal since that transaction.
These actions are what Bitcoin wallet software does for you.
Outside your wallet you need to make regular backups and keep a separate copy of your secret "private key" numbers somewhere safe. Alternatively some wallets give you a recovery phrase (seed phrase) which is a set of words which can be used to re-create the secret numbers. This should be kept somewhere safe in case your computer or device is lost, stolen or broken.

To check the amounts programatically without using your wallet, you'd need to write a program to do every essential thing the wallet does.
One way to programatically and independently check amounts would be to install some different wallet software on a different device and import your Bitcoin addresses as "watch-only" addresses. This other wallet would calculate amounts controlled by the secret numbers from which those Bitcoin addresses were calculated. This other wallet could not be used to spend money.


text:
How to determine which private key signed a transaction
I was just looking through some of the blocks and I saw a transaction which had 408.1011 BTC split into 3 output address. 18.5468, 230.6512 and 158.9030731.
I then saw the 230.6512 use the transaction id and send it to 2 address. I know it was the 230.6512 only because the output was 9.2293 & 221.420555.
My question is, how can I find out that the address associated with 230.6512 is spent but the other 2 are not?
The reason I'm asking is because if you don't know which private key signed this transaction, then how can you find out how much balance a private key has left to spend?
Edit: Just found vout Looks like this is what I was looking for. Seems that it points to the index of the output used.

Your question is based on a misconception. The Bitcoin protocol does not operate on addresses, or balances.
Transactions have outputs and inputs. Each outputs creates a "coin", of a certain value, and with a certain condition under which it can be spent. Every input spends a coin - in its entirety - by referencing that coin explicitly, and proving it is allowed to spend it.
Those spending conditions are often represented for human consumption as addresses, but "address balance" is simply the sum of the values of all coins with a spending condition matching that address, which have not yet been spent.
So how do wallets, and the protocol, determine whether a transaction is valid? Not by looking at its balance, but by checking that the coin referenced in the input hasn't been spent yet, that the transaction contains valid signature(s) for it, and that the sum of the values of the inputs is at least as much as the value of the outputs being created.
What block explorer websites show you is deceiving. They try to represent transactions as reducing the balance of some addresses, and incrementing the balance of others. This isn't what's actually happens. In reality, transactions just melt down and reforge coins.


text:
How can Segwit increase transaction throughput if the same amount of data is stored in the blockchain?
I'm posting this after reading the answers to this: Where is the Signature stored in a SegWit transaction?
I've found out from reading that, that in Segwit, the "witnesses" (or signatures) that verify a transaction's authenticity are still stored on the blockchain. In which case, the same amount of bits ought to be used up by a given transaction as if Segwit weren't being used. If the block size remains the same, let's say 1MB, then it ought to hold the exact same number of transactions.
I've got an alternative possibility. Let's say that what actually gets sent to the nodes is a transaction with an empty signatures field (I think that such a field doesn't quite exist because of multi-sig stuff and scripts, but bear with me). Then the signatures can be sent through a separate channel. What actually gets stamped onto the blockchain is a transaction with an empty signatures field. And then because of the longest-blockchain-always-wins rule, the non-Segwit nodes will simply accept that the blockchain with the most number of transactions, which includes some funny signature-less transactions, is the authoritative blockchain.
But apparently reality is more complicated.
Also, there's a question about how much the longest-blockchain-always-wins rule can be abused. (The word "abused" does not imply a negative opinion). For instance, let's say I come up with FoobarWit. FoobarWit is the same as Bitcoin but with a larger blocksize (no Segwit). Once enough nodes adopt FoobarWit, then more such nodes will win the mining race, and so other nodes will effectively accept bigger blocks because the history is longer. I know that's not the case, but why?

related: https://bitcoin.stackexchange.com/q/52196/5406

@Murch I read that. Didn't really help

The answer to the question "Are the segwit witnesses part of the blockchain" depends on what you define as the blockchain:

According to old pre-segwit nodes, the answer is no, as they don't care or receive the witnesses.
According to new segwit nodes, the answer is yes; the witnesses are as much part of the chain as everything else, and subject to just as many validation rules.

So segwit increases throughput slightly, at the cost of increasing the size of the blockchain. There is no magic bullet here, and this aspect of segwit is simply a block size increase. It is not a scalability improvement.
Segwit was primarily a solution for the transaction malleability problem, opening the door for various higher-level protocols that were handicapped by being unable to reason about future on-chain transactions that weren't published yet. In addition it also improved he incentive structure a bit (making spending relatively cheaper w.r.t. creating outputs). As a final side effect of that, it also increased the block size a bit in a backward compatible way.
As for your question about another fork that goes even further: what your software does has no effect on those who don't adopt it. Even if it has all the hashrate in the world on board, old nodes won't care or see your additional data.

But what makes this change able to not trigger a hard fork? And how can something both be and not be on the blockchain? I thought there was only one chain.

I can't just take what you say on faith

The only real explanation would be a story of a segwit transaction between Alice and Bob making it through the network. Everything else is "It magically can do this!" Though James C.'s explanation is starting to help

I'm reading the BIP. If I have a breakthrough, I'll post an answer

Segwit nodes simply strip out the witnesses when relaying to non-segwit peers. As the witnesses do not affect the txid, there is nothing in a transaction that alludes or refers to its witness in any way, so nothing is broken for nodes who don't care about those witnesses. However, the block does contain a hash of all witnesses (in a way that looks like a piece of random data to old nodes), making it part of the chain for those who care about it.

It helps to realize that the blockchain is not just a blob of bytes. It's a structure with pieces of data that refers to other pieces of data with hashes. Whether a node requires the data pointed to by those hashes is its own decision (for example, light nodes have the headers but don't require the transactions). Segwit simply added a new piece of data, which new nodes care about and others don't.

So it's almost like saying that we make a backwards-compatible dialect of C where some comments (which older compilers would ignore) become executable. I kind of get that. But then you're saying that the block contains only the hash of the witnesses

My understanding is that hashing is an irreversible operation. So you only put a scrambled version of the witness data onto the blockchain

It's wrong to think of blocks to "contain" anything. They're all just hashes. In the same way you can also say transactions are not "part" of the blockchain, because block headers just contain a hash of the transactions. Whether you think of transactions being an essential part of the chain depends on the context: full nodes do (they won't accept a block with tx hashes without the actual transactions), light nodes don't (they see the hashes and are happy with that). The same is true for segwit witnesses.

What is the point of a light node then? What on Earth can it do? I'm lost

It can verify that transactions were included in the chain, selectively, without verifying that their signatures are valid. This is not as secure as a full node, but assuming enough others do run full node, it has much better usability with a security level that's acceptable to some use cases. Note that I'm not saying that segwit is related to this security tradeoff (it isn't), but it's a similar principle. You need to stop thinking about the chain being just a blob of data that contains some things and not other things. It's a structure with commitments, which some care about.

The 1MB block limit still stands after segwit activation, but it solely refers to the block transactions serialised in the pre-segwit format, which includes no witnesses. This 1MB is still consensus today.
With segwit activated, there is an additional block size constraint, which DOES apply to segwit data: It is a "weight" limit of 4M per block.
The weight is computed as follows:

Transaction Bytes (Pre-segwit serialisation) x 3
Plus Transaction Bytes with Segwit (Segwit serialisation) x 1

As the witness portion of the total transaction data increases towards 100%, the effective block size limit in Bytes approaches 4MB.

So the witnesses are not stamped onto the blockchain?

The block size constraint is to avoid DDOSes via the Segwit channel?

Good question: they are committed as witness root(like the merkleroot) together with the coinbase witness in a coinbase op_return output, which is validated for correctness.

Does the op_return field not count towards the classical 1MB block size?

There is no separate channel. If a node is not segwit ready, peers recognize this on the P2P level and refrain from sending witness data to the older peer. Witness data is “stripped” from the tx data.

Yes op_return is not witness data and therefore contributes to the 1MB block limit.

"committed as witness root". Onto the blockchain? Thanks.

Yes in the coinbase transaction, which is the first tx in every block :) you can find details in one of the segwit BIPs. Something like dsha256(witnessroot|coinbasewitness).

Let us continue this discussion in chat.