Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Empirical Explorations in Training Networks with Discrete Activations   We present extensive experiments training and testing hidden units in deep
networks that emit only a predefined, static, number of discretized values.
These units provide benefits in real-world deployment in systems in which
memory and/or computation may be limited. Additionally, they are particularly
well suited for use in large recurrent network models that require the
maintenance of large amounts of internal state in memory. Surprisingly, we find
that despite reducing the number of values that can be represented in the
output activations from $2^{32}-2^{64}$ to between 64 and 256, there is little
to no degradation in network performance across a variety of different
settings. We investigate simple classification and regression tasks, as well as
memorization and compression problems. We compare the results with more
standard activations, such as tanh and relu. Unlike previous discretization
studies which often concentrate only on binary units, we examine the effects of
varying the number of allowed activation levels. Compared to existing
approaches for discretization, the approach presented here is both conceptually
and programatically simple, has no stochastic component, and allows the
training, testing, and usage phases to be treated in exactly the same manner.

10
The categories of this paper are: q-bio
18
The categories of this paper are: econ
14
The categories of this paper are: math-ph
29
The categories of this paper are: funct-an
3
The categories of this paper are: physics
16
The categories of this paper are: stat
32
The categories of this paper are: atom-ph
15
The categories of this paper are: nucl-ex
25
The categories of this paper are: patt-sol
36
The categories of this paper are: cmp-lg
23
The categories of this paper are: solv-int
30
The categories of this paper are: q-alg
27
The categories of this paper are: comp-gas
8
The categories of this paper are: hep-ex
35
The categories of this paper are: mtrl-th
4
The categories of this paper are: cond-mat