Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Learning to Rank from Relevance Judgments Distributions   Learning to Rank (LETOR) algorithms are usually trained on annotated corpora
where a single relevance label is assigned to each available document-topic
pair. Within the Cranfield framework, relevance labels result from merging
either multiple expertly curated or crowdsourced human assessments. In this
paper, we explore how to train LETOR models with relevance judgments
distributions (either real or synthetically generated) assigned to
document-topic pairs instead of single-valued relevance labels. We propose five
new probabilistic loss functions to deal with the higher expressive power
provided by relevance judgments distributions and show how they can be applied
both to neural and GBM architectures. Moreover, we show how training a LETOR
model on a sampled version of the relevance judgments from certain probability
distributions can improve its performance when relying either on traditional or
probabilistic loss functions. Finally, we validate our hypothesis on real-world
crowdsourced relevance judgments distributions. Overall, we observe that
relying on relevance judgments distributions to train different LETOR models
can boost their performance and even outperform strong baselines such as
LambdaMART on several test collections.

37
The categories of this paper are: supr-con
19
The categories of this paper are: eess
23
The categories of this paper are: solv-int
20
The categories of this paper are: acc-phys
17
The categories of this paper are: q-fin
11
The categories of this paper are: hep-lat
8
The categories of this paper are: hep-ex
26
The categories of this paper are: bayes-an
21
The categories of this paper are: adap-org
34
The categories of this paper are: chem-ph
5
The categories of this paper are: gr-qc
2
The categories of this paper are: math
29
The categories of this paper are: funct-an
15
The categories of this paper are: nucl-ex
22
The categories of this paper are: chao-dyn
31
The categories of this paper are: ao-sci