Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Modelling General Properties of Nouns by Selectively Averaging
  Contextualised Embeddings   While the success of pre-trained language models has largely eliminated the
need for high-quality static word vectors in many NLP applications, such
vectors continue to play an important role in tasks where words need to be
modelled in the absence of linguistic context. In this paper, we explore how
the contextualised embeddings predicted by BERT can be used to produce
high-quality word vectors for such domains, in particular related to knowledge
base completion, where our focus is on capturing the semantic properties of
nouns. We find that a simple strategy of averaging the contextualised
embeddings of masked word mentions leads to vectors that outperform the static
word vectors learned by BERT, as well as those from standard word embedding
models, in property induction tasks. We notice in particular that masking
target words is critical to achieve this strong performance, as the resulting
vectors focus less on idiosyncratic properties and more on general semantic
properties. Inspired by this view, we propose a filtering strategy which is
aimed at removing the most idiosyncratic mention vectors, allowing us to obtain
further performance gains in property induction.

31
The categories of this paper are: ao-sci
22
The categories of this paper are: chao-dyn
20
The categories of this paper are: acc-phys
24
The categories of this paper are: dg-ga
36
The categories of this paper are: cmp-lg
2
The categories of this paper are: math
3
The categories of this paper are: physics
30
The categories of this paper are: q-alg
4
The categories of this paper are: cond-mat
12
The categories of this paper are: nucl-th
10
The categories of this paper are: q-bio
37
The categories of this paper are: supr-con
26
The categories of this paper are: bayes-an
29
The categories of this paper are: funct-an
34
The categories of this paper are: chem-ph
27
The categories of this paper are: comp-gas