Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Enhancing Simple Models by Exploiting What They Already Know   There has been recent interest in improving performance of simple models for
multiple reasons such as interpretability, robust learning from small data,
deployment in memory constrained settings as well as environmental
considerations. In this paper, we propose a novel method SRatio that can
utilize information from high performing complex models (viz. deep neural
networks, boosted trees, random forests) to reweight a training dataset for a
potentially low performing simple model of much lower complexity such as a
decision tree or a shallow network enhancing its performance. Our method also
leverages the per sample hardness estimate of the simple model which is not the
case with the prior works which primarily consider the complex model's
confidences/predictions and is thus conceptually novel. Moreover, we generalize
and formalize the concept of attaching probes to intermediate layers of a
neural network to other commonly used classifiers and incorporate this into our
method. The benefit of these contributions is witnessed in the experiments
where on 6 UCI datasets and CIFAR-10 we outperform competitors in a majority
(16 out of 27) of the cases and tie for best performance in the remaining
cases. In fact, in a couple of cases, we even approach the complex model's
performance. We also conduct further experiments to validate assertions and
intuitively understand why our method works. Theoretically, we motivate our
approach by showing that the weighted loss minimized by simple models using our
weighting upper bounds the loss of the complex model.

18
The categories of this paper are: econ
34
The categories of this paper are: chem-ph
11
The categories of this paper are: hep-lat
28
The categories of this paper are: alg-geom
31
The categories of this paper are: ao-sci
32
The categories of this paper are: atom-ph
22
The categories of this paper are: chao-dyn
23
The categories of this paper are: solv-int
15
The categories of this paper are: nucl-ex
26
The categories of this paper are: bayes-an
4
The categories of this paper are: cond-mat
21
The categories of this paper are: adap-org
13
The categories of this paper are: quant-ph
19
The categories of this paper are: eess
33
The categories of this paper are: plasm-ph
25
The categories of this paper are: patt-sol