Not able to Reproduce results in MTEB
Hello,
I am trying to integrate Qwen3-reranker models(0.6B, 4B, 8B) in MTEB. But we were not able to reproduce the reported results. 1 of the main reasons is that the reported results are doing reranking from Qwen3-embeddings, which we were not able to reproduce. Can any of the co-authors please check the PR in MTEB, mainly if its implementation is correct or not? Its opened from a long time, and we tagged you, but you seem to have missed it.
PR Link: https://github.com/embeddings-benchmark/mteb/pull/3958
Looks like the changes introduced 12-days back were breaking. Our code failed as well.
We have merged PR with the implementation in MTEB: https://github.com/embeddings-benchmark/mteb/pull/3958, though we were not able to reproduce scores. Though any of the co-authors want to fix it, they are welcome to open a PR in MTEB!
I am closing it for now.