Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Approximating Interactive Human Evaluation with Self-Play for
  Open-Domain Dialog Systems   Building an open-domain conversational agent is a challenging problem.
Current evaluation methods, mostly post-hoc judgments of static conversation,
do not capture conversation quality in a realistic interactive context. In this
paper, we investigate interactive human evaluation and provide evidence for its
necessity; we then introduce a novel, model-agnostic, and dataset-agnostic
method to approximate it. In particular, we propose a self-play scenario where
the dialog system talks to itself and we calculate a combination of proxies
such as sentiment and semantic coherence on the conversation trajectory. We
show that this metric is capable of capturing the human-rated quality of a
dialog model better than any automated metric known to-date, achieving a
significant Pearson correlation (r>.7, p<.05). To investigate the strengths of
this novel metric and interactive evaluation in comparison to state-of-the-art
metrics and human evaluation of static conversations, we perform extended
experiments with a set of models, including several that make novel
improvements to recent hierarchical dialog generation architectures through
sentiment and semantic knowledge distillation on the utterance level. Finally,
we open-source the interactive evaluation platform we built and the dataset we
collected to allow researchers to efficiently deploy and evaluate dialog
models.

12
The categories of this paper are: nucl-th
5
The categories of this paper are: gr-qc
14
The categories of this paper are: math-ph
31
The categories of this paper are: ao-sci
3
The categories of this paper are: physics
34
The categories of this paper are: chem-ph
32
The categories of this paper are: atom-ph
23
The categories of this paper are: solv-int
22
The categories of this paper are: chao-dyn
33
The categories of this paper are: plasm-ph
4
The categories of this paper are: cond-mat
9
The categories of this paper are: nlin
21
The categories of this paper are: adap-org
17
The categories of this paper are: q-fin
28
The categories of this paper are: alg-geom
19
The categories of this paper are: eess