Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Harmony: Overcoming the hurdles of GPU memory capacity to train massive
  DNN models on commodity servers   Deep neural networks (DNNs) have grown exponentially in complexity and size
over the past decade, leaving only those who have access to massive
datacenter-based resources with the ability to develop and train such models.
One of the main challenges for the long tail of researchers who might have
access to only limited resources (e.g., a single multi-GPU server) is limited
GPU memory capacity compared to model size. The problem is so acute that the
memory requirement of training large DNN models can often exceed the aggregate
capacity of all available GPUs on commodity servers; this problem only gets
worse with the trend of ever-growing model sizes. Current solutions that rely
on virtualizing GPU memory (by swapping to/from CPU memory) incur excessive
swapping overhead. In this paper, we present a new training framework, Harmony,
and advocate rethinking how DNN frameworks schedule computation and move data
to push the boundaries of training large models efficiently on modest multi-GPU
deployments. Across many large DNN models, Harmony is able to reduce swap load
by up to two orders of magnitude and obtain a training throughput speedup of up
to 7.6x over highly optimized baselines with virtualized memory.

30
The categories of this paper are: q-alg
19
The categories of this paper are: eess
18
The categories of this paper are: econ
29
The categories of this paper are: funct-an
25
The categories of this paper are: patt-sol
16
The categories of this paper are: stat
8
The categories of this paper are: hep-ex
28
The categories of this paper are: alg-geom
9
The categories of this paper are: nlin
24
The categories of this paper are: dg-ga
2
The categories of this paper are: math
14
The categories of this paper are: math-ph
0
The categories of this paper are: hep-ph
4
The categories of this paper are: cond-mat
12
The categories of this paper are: nucl-th
23
The categories of this paper are: solv-int