Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Variance Reduction in SGD by Distributed Importance Sampling   Humans are able to accelerate their learning by selecting training materials
that are the most informative and at the appropriate level of difficulty. We
propose a framework for distributing deep learning in which one set of workers
search for the most informative examples in parallel while a single worker
updates the model on examples selected by importance sampling. This leads the
model to update using an unbiased estimate of the gradient which also has
minimum variance when the sampling proposal is proportional to the L2-norm of
the gradient. We show experimentally that this method reduces gradient variance
even in a context where the cost of synchronization across machines cannot be
ignored, and where the factors for importance sampling are not updated
instantly across the training set.

33
The categories of this paper are: plasm-ph
22
The categories of this paper are: chao-dyn
13
The categories of this paper are: quant-ph
3
The categories of this paper are: physics
12
The categories of this paper are: nucl-th
37
The categories of this paper are: supr-con
15
The categories of this paper are: nucl-ex
14
The categories of this paper are: math-ph
32
The categories of this paper are: atom-ph
21
The categories of this paper are: adap-org
11
The categories of this paper are: hep-lat
34
The categories of this paper are: chem-ph
5
The categories of this paper are: gr-qc
6
The categories of this paper are: astro-ph
20
The categories of this paper are: acc-phys
36
The categories of this paper are: cmp-lg