txscript: Rework check pubkey encoding test.
txscript: Rework check pubkey encoding test.

This reworks the tests for CheckPubKeyEncoding to test for the explicit
reason for the failure to ensure that each test is actually testing the
intended condition and be more consistent with the more recent tests.

It also adds several more tests to better ensure proper behavior and
includes a couple of corner cases where public keys will ultimately
fail validation due to not being on the curve, but are still valid
encodings from the standpoint of preventing malleability.

decred/dcrd,decred/dcrd