Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: What is the ground truth? Reliability of multi-annotator data for audio
  tagging   Crowdsourcing has become a common approach for annotating large amounts of
data. It has the advantage of harnessing a large workforce to produce large
amounts of data in a short time, but comes with the disadvantage of employing
non-expert annotators with different backgrounds. This raises the problem of
data reliability, in addition to the general question of how to combine the
opinions of multiple annotators in order to estimate the ground truth. This
paper presents a study of the annotations and annotators' reliability for audio
tagging. We adapt the use of Krippendorf's alpha and multi-annotator competence
estimation (MACE) for a multi-labeled data scenario, and present how MACE can
be used to estimate a candidate ground truth based on annotations from
non-expert users with different levels of expertise and competence.

18
The categories of this paper are: econ
37
The categories of this paper are: supr-con
31
The categories of this paper are: ao-sci
32
The categories of this paper are: atom-ph
13
The categories of this paper are: quant-ph
17
The categories of this paper are: q-fin
21
The categories of this paper are: adap-org
1
The categories of this paper are: cs
33
The categories of this paper are: plasm-ph
26
The categories of this paper are: bayes-an
36
The categories of this paper are: cmp-lg
14
The categories of this paper are: math-ph
23
The categories of this paper are: solv-int
4
The categories of this paper are: cond-mat
0
The categories of this paper are: hep-ph
8
The categories of this paper are: hep-ex