Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Better Long-Range Dependency By Bootstrapping A Mutual Information
  Regularizer   In this work, we develop a novel regularizer to improve the learning of
long-range dependency of sequence data. Applied on language modelling, our
regularizer expresses the inductive bias that sequence variables should have
high mutual information even though the model might not see abundant
observations for complex long-range dependency. We show how the `next sentence
prediction (classification)' heuristic can be derived in a principled way from
our mutual information estimation framework, and be further extended to
maximize the mutual information of sequence variables. The proposed approach
not only is effective at increasing the mutual information of segments under
the learned model but more importantly, leads to a higher likelihood on holdout
data, and improved generation quality. Code is released at
https://github.com/BorealisAI/BMI.

13
The categories of this paper are: quant-ph
22
The categories of this paper are: chao-dyn
37
The categories of this paper are: supr-con
35
The categories of this paper are: mtrl-th
7
The categories of this paper are: hep-th
21
The categories of this paper are: adap-org
12
The categories of this paper are: nucl-th
27
The categories of this paper are: comp-gas
33
The categories of this paper are: plasm-ph
5
The categories of this paper are: gr-qc
25
The categories of this paper are: patt-sol
36
The categories of this paper are: cmp-lg
34
The categories of this paper are: chem-ph
17
The categories of this paper are: q-fin
10
The categories of this paper are: q-bio
15
The categories of this paper are: nucl-ex