Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Zero-Cost, Arrow-Enabled Data Interface for Apache Spark   Distributed data processing ecosystems are widespread and their components
are highly specialized, such that efficient interoperability is urgent.
Recently, Apache Arrow was chosen by the community to serve as a format
mediator, providing efficient in-memory data representation. Arrow enables
efficient data movement between data processing and storage engines,
significantly improving interoperability and overall performance. In this work,
we design a new zero-cost data interoperability layer between Apache Spark and
Arrow-based data sources through the Arrow Dataset API. Our novel data
interface helps separate the computation (Spark) and data (Arrow) layers. This
enables practitioners to seamlessly use Spark to access data from all Arrow
Dataset API-enabled data sources and frameworks. To benefit our community, we
open-source our work and show that consuming data through Apache Arrow is
zero-cost: our novel data interface is either on-par or more performant than
native Spark.

8
The categories of this paper are: hep-ex
26
The categories of this paper are: bayes-an
2
The categories of this paper are: math
32
The categories of this paper are: atom-ph
12
The categories of this paper are: nucl-th
37
The categories of this paper are: supr-con
0
The categories of this paper are: hep-ph
11
The categories of this paper are: hep-lat
35
The categories of this paper are: mtrl-th
20
The categories of this paper are: acc-phys
29
The categories of this paper are: funct-an
3
The categories of this paper are: physics
15
The categories of this paper are: nucl-ex
16
The categories of this paper are: stat
25
The categories of this paper are: patt-sol
6
The categories of this paper are: astro-ph