Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Making Coherence Out of Nothing At All: Measuring the Evolution of
  Gradient Alignment   We propose a new metric ($m$-coherence) to experimentally study the alignment
of per-example gradients during training. Intuitively, given a sample of size
$m$, $m$-coherence is the number of examples in the sample that benefit from a
small step along the gradient of any one example on average. We show that
compared to other commonly used metrics, $m$-coherence is more interpretable,
cheaper to compute ($O(m)$ instead of $O(m^2)$) and mathematically cleaner. (We
note that $m$-coherence is closely connected to gradient diversity, a quantity
previously used in some theoretical bounds.) Using $m$-coherence, we study the
evolution of alignment of per-example gradients in ResNet and Inception models
on ImageNet and several variants with label noise, particularly from the
perspective of the recently proposed Coherent Gradients (CG) theory that
provides a simple, unified explanation for memorization and generalization
[Chatterjee, ICLR 20]. Although we have several interesting takeaways, our most
surprising result concerns memorization. Naively, one might expect that when
training with completely random labels, each example is fitted independently,
and so $m$-coherence should be close to 1. However, this is not the case:
$m$-coherence reaches much higher values during training (100s), indicating
that over-parameterized neural networks find common patterns even in scenarios
where generalization is not possible. A detailed analysis of this phenomenon
provides both a deeper confirmation of CG, but at the same point puts into
sharp relief what is missing from the theory in order to provide a complete
explanation of generalization in neural networks.

34
The categories of this paper are: chem-ph
27
The categories of this paper are: comp-gas
21
The categories of this paper are: adap-org
15
The categories of this paper are: nucl-ex
33
The categories of this paper are: plasm-ph
29
The categories of this paper are: funct-an
6
The categories of this paper are: astro-ph
8
The categories of this paper are: hep-ex
5
The categories of this paper are: gr-qc
10
The categories of this paper are: q-bio
20
The categories of this paper are: acc-phys
35
The categories of this paper are: mtrl-th
7
The categories of this paper are: hep-th
3
The categories of this paper are: physics
37
The categories of this paper are: supr-con
9
The categories of this paper are: nlin