Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: The challenge of reproducible ML: an empirical study on the impact of
  bugs   Reproducibility is a crucial requirement in scientific research. When results
of research studies and scientific papers have been found difficult or
impossible to reproduce, we face a challenge which is called reproducibility
crisis. Although the demand for reproducibility in Machine Learning (ML) is
acknowledged in the literature, a main barrier is inherent non-determinism in
ML training and inference. In this paper, we establish the fundamental factors
that cause non-determinism in ML systems. A framework, ReproduceML, is then
introduced for deterministic evaluation of ML experiments in a real, controlled
environment. ReproduceML allows researchers to investigate software
configuration effects on ML training and inference. Using ReproduceML, we run a
case study: investigation of the impact of bugs inside ML libraries on
performance of ML experiments. This study attempts to quantify the impact that
the occurrence of bugs in a popular ML framework, PyTorch, has on the
performance of trained models. To do so, a comprehensive methodology is
proposed to collect buggy versions of ML libraries and run deterministic ML
experiments using ReproduceML. Our initial finding is that there is no evidence
based on our limited dataset to show that bugs which occurred in PyTorch do
affect the performance of trained models. The proposed methodology as well as
ReproduceML can be employed for further research on non-determinism and bugs.

15
The categories of this paper are: nucl-ex
10
The categories of this paper are: q-bio
32
The categories of this paper are: atom-ph
21
The categories of this paper are: adap-org
31
The categories of this paper are: ao-sci
27
The categories of this paper are: comp-gas
24
The categories of this paper are: dg-ga
13
The categories of this paper are: quant-ph
33
The categories of this paper are: plasm-ph
4
The categories of this paper are: cond-mat
29
The categories of this paper are: funct-an
30
The categories of this paper are: q-alg
3
The categories of this paper are: physics
20
The categories of this paper are: acc-phys
25
The categories of this paper are: patt-sol
36
The categories of this paper are: cmp-lg