Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Meta-evaluation of Conversational Search Evaluation Metrics   Conversational search systems, such as Google Assistant and Microsoft
Cortana, enable users to interact with search systems in multiple rounds
through natural language dialogues. Evaluating such systems is very challenging
given that any natural language responses could be generated, and users
commonly interact for multiple semantically coherent rounds to accomplish a
search task. Although prior studies proposed many evaluation metrics, the
extent of how those measures effectively capture user preference remains to be
investigated. In this paper, we systematically meta-evaluate a variety of
conversational search metrics. We specifically study three perspectives on
those metrics: (1) reliability: the ability to detect "actual" performance
differences as opposed to those observed by chance; (2) fidelity: the ability
to agree with ultimate user preference; and (3) intuitiveness: the ability to
capture any property deemed important: adequacy, informativeness, and fluency
in the context of conversational search. By conducting experiments on two test
collections, we find that the performance of different metrics varies
significantly across different scenarios whereas consistent with prior studies,
existing metrics only achieve a weak correlation with ultimate user preference
and satisfaction. METEOR is, comparatively speaking, the best existing
single-turn metric considering all three perspectives. We also demonstrate that
adapted session-based evaluation metrics can be used to measure multi-turn
conversational search, achieving moderate concordance with user satisfaction.
To our knowledge, our work establishes the most comprehensive meta-evaluation
for conversational search to date.

12
The categories of this paper are: nucl-th
22
The categories of this paper are: chao-dyn
20
The categories of this paper are: acc-phys
35
The categories of this paper are: mtrl-th
24
The categories of this paper are: dg-ga
18
The categories of this paper are: econ
15
The categories of this paper are: nucl-ex
3
The categories of this paper are: physics
28
The categories of this paper are: alg-geom
2
The categories of this paper are: math
10
The categories of this paper are: q-bio
32
The categories of this paper are: atom-ph
26
The categories of this paper are: bayes-an
19
The categories of this paper are: eess
16
The categories of this paper are: stat
30
The categories of this paper are: q-alg