Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Back to Square One: Artifact Detection, Training and Commonsense
  Disentanglement in the Winograd Schema   The Winograd Schema (WS) has been proposed as a test for measuring
commonsense capabilities of models. Recently, pre-trained language model-based
approaches have boosted performance on some WS benchmarks but the source of
improvement is still not clear. This paper suggests that the apparent progress
on WS may not necessarily reflect progress in commonsense reasoning. To support
this claim, we first show that the current evaluation method of WS is
sub-optimal and propose a modification that uses twin sentences for evaluation.
We also propose two new baselines that indicate the existence of artifacts in
WS benchmarks. We then develop a method for evaluating WS-like sentences in a
zero-shot setting to account for the commonsense reasoning abilities acquired
during the pretraining and observe that popular language models perform
randomly in this setting when using our more strict evaluation. We conclude
that the observed progress is mostly due to the use of supervision in training
WS models, which is not likely to successfully support all the required
commonsense reasoning skills and knowledge.

14
The categories of this paper are: math-ph
29
The categories of this paper are: funct-an
27
The categories of this paper are: comp-gas
36
The categories of this paper are: cmp-lg
13
The categories of this paper are: quant-ph
3
The categories of this paper are: physics
8
The categories of this paper are: hep-ex
25
The categories of this paper are: patt-sol
37
The categories of this paper are: supr-con
24
The categories of this paper are: dg-ga
30
The categories of this paper are: q-alg
11
The categories of this paper are: hep-lat
31
The categories of this paper are: ao-sci
19
The categories of this paper are: eess
32
The categories of this paper are: atom-ph
34
The categories of this paper are: chem-ph