Verify JxlPixelFormat validity when setting out callback
Verify JxlPixelFormat validity when setting out callback

The format must be checked for having a supported amount of channels,
pixel type, etc... This was done in the size check for the image output
buffer setters, but was not done in the callback because buffer size
works differently for those.

Now the full check is done here as well to fix that.

libjxl/libjxl,libjxl/libjxl,libjxl/libjxl,libjxl/libjxl,libjxl/libjxl,libjxl/libjxl,libjxl/libjxl,libjxl/libjxl