Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Continual Normalization: Rethinking Batch Normalization for Online
  Continual Learning   Existing continual learning methods use Batch Normalization (BN) to
facilitate training and improve generalization across tasks. However, the
non-i.i.d and non-stationary nature of continual learning data, especially in
the online setting, amplify the discrepancy between training and testing in BN
and hinder the performance of older tasks. In this work, we study the
cross-task normalization effect of BN in online continual learning where BN
normalizes the testing data using moments biased towards the current task,
resulting in higher catastrophic forgetting. This limitation motivates us to
propose a simple yet effective method that we call Continual Normalization (CN)
to facilitate training similar to BN while mitigating its negative effect.
Extensive experiments on different continual learning algorithms and online
scenarios show that CN is a direct replacement for BN and can provide
substantial performance improvements. Our implementation is available at
\url{https://github.com/phquang/Continual-Normalization}.

35
The categories of this paper are: mtrl-th
37
The categories of this paper are: supr-con
30
The categories of this paper are: q-alg
2
The categories of this paper are: math
29
The categories of this paper are: funct-an
14
The categories of this paper are: math-ph
11
The categories of this paper are: hep-lat
16
The categories of this paper are: stat
7
The categories of this paper are: hep-th
15
The categories of this paper are: nucl-ex
23
The categories of this paper are: solv-int
8
The categories of this paper are: hep-ex
32
The categories of this paper are: atom-ph
18
The categories of this paper are: econ
22
The categories of this paper are: chao-dyn
3
The categories of this paper are: physics