Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: An Information Theory-inspired Strategy for Automatic Network Pruning   Despite superior performance on many computer vision tasks, deep convolution
neural networks are well known to be compressed on devices that have resource
constraints. Most existing network pruning methods require laborious human
efforts and prohibitive computation resources, especially when the constraints
are changed. This practically limits the application of model compression when
the model needs to be deployed on a wide range of devices. Besides, existing
methods are still challenged by the missing theoretical guidance. In this paper
we propose an information theory-inspired strategy for automatic model
compression. The principle behind our method is the information bottleneck
theory, i.e., the hidden representation should compress information with each
other. We thus introduce the normalized Hilbert-Schmidt Independence Criterion
(nHSIC) on network activations as a stable and generalized indicator of layer
importance. When a certain resource constraint is given, we integrate the HSIC
indicator with the constraint to transform the architecture search problem into
a linear programming problem with quadratic constraints. Such a problem is
easily solved by a convex optimization method with a few seconds. We also
provide a rigorous proof to reveal that optimizing the normalized HSIC
simultaneously minimizes the mutual information between different layers.
Without any search process, our method achieves better compression tradeoffs
comparing to the state-of-the-art compression algorithms. For instance, with
ResNet-50, we achieve a 45.3%-FLOPs reduction, with a 75.75 top-1 accuracy on
ImageNet. Codes are avaliable at
https://github.com/MAC-AutoML/ITPruner/tree/master.

9
The categories of this paper are: nlin
24
The categories of this paper are: dg-ga
29
The categories of this paper are: funct-an
14
The categories of this paper are: math-ph
31
The categories of this paper are: ao-sci
37
The categories of this paper are: supr-con
34
The categories of this paper are: chem-ph
35
The categories of this paper are: mtrl-th
26
The categories of this paper are: bayes-an
25
The categories of this paper are: patt-sol
15
The categories of this paper are: nucl-ex
10
The categories of this paper are: q-bio
2
The categories of this paper are: math
6
The categories of this paper are: astro-ph
0
The categories of this paper are: hep-ph
18
The categories of this paper are: econ