Batch inference
I noticed that when doing batch inference, if the images in the batch are roughly of the same size, the model output similar result to single inference. But when the sizes are different, the result for the smaller images always look worse than single inference result. Could this be because of padding tokens? Which padding token is recommended for batch inference?
We used the same padding token mentioned in the config here. Don't think it will be padding token issue unless you are sending some other token manually.
No, I'm using the default padding token. Could you think of any other reason that make batch inference perform worse than single inference? Thank you
Batch inference should perform the same as single inference. Do you have any reproducible code? I can look into it.