Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Scaling GRPC Tensorflow on 512 nodes of Cori Supercomputer   We explore scaling of the standard distributed Tensorflow with GRPC
primitives on up to 512 Intel Xeon Phi (KNL) nodes of Cori supercomputer with
synchronous stochastic gradient descent (SGD), and identify causes of scaling
inefficiency at higher node counts. To our knowledge, this is the first
exploration of distributed GRPC Tensorflow scalability on a HPC supercomputer
at such large scale with synchronous SGD. We studied scaling of two convolution
neural networks - ResNet-50, a state-of-the-art deep network for classification
with roughly 25.5 million parameters, and HEP-CNN, a shallow topology with less
than 1 million parameters for common scientific usages. For ResNet-50, we
achieve >80% scaling efficiency on up to 128 workers, using 32 parameter
servers (PS tasks) with a steep decline down to 23% for 512 workers using 64 PS
tasks. Our analysis of the efficiency drop points to low network bandwidth
utilization due to combined effect of three factors. (a) Heterogeneous
distributed parallelization algorithm which uses PS tasks as centralized
servers for gradient averaging is suboptimal for utilizing interconnect
bandwidth. (b) Load imbalance among PS tasks hinders their efficient scaling.
(c) Underlying communication primitive GRPC is currently inefficient on Cori
high-speed interconnect. The HEP-CNN demands less interconnect bandwidth, and
shows >80% weak scaling efficiency for up to 256 nodes with only 1 PS task. Our
findings are applicable to other deep learning networks. Big networks with
millions of parameters stumble upon the issues discussed here. Shallower
networks like HEP-CNN with relatively lower number of parameters can
efficiently enjoy weak scaling even with a single parameter server.

31
The categories of this paper are: ao-sci
21
The categories of this paper are: adap-org
10
The categories of this paper are: q-bio
24
The categories of this paper are: dg-ga
32
The categories of this paper are: atom-ph
25
The categories of this paper are: patt-sol
28
The categories of this paper are: alg-geom
34
The categories of this paper are: chem-ph
35
The categories of this paper are: mtrl-th
16
The categories of this paper are: stat
3
The categories of this paper are: physics
23
The categories of this paper are: solv-int
26
The categories of this paper are: bayes-an
7
The categories of this paper are: hep-th
22
The categories of this paper are: chao-dyn
11
The categories of this paper are: hep-lat