Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Programming Puzzles   We introduce a new type of programming challenge called programming puzzles,
as an objective and comprehensive evaluation of program synthesis, and release
an open-source dataset of Python Programming Puzzles (P3). Each puzzle is
defined by a short Python program $f$, and the goal is to find an input which
makes $f$ return True. The puzzles are objective in that each one is specified
entirely by the source code of its verifier $f$, so evaluating $f$ is all that
is needed to test a candidate solution. They do not require an answer key or
input/output examples, nor do they depend on natural language understanding.
The dataset is comprehensive in that it spans problems of a range of
difficulties and domains, ranging from trivial string manipulation problems, to
classic programming puzzles (e.g., Tower of Hanoi), to
interview/competitive-programming problems (e.g., dynamic programming), to
longstanding open problems in algorithms and mathematics (e.g., factoring). We
develop baseline enumerative program synthesis, GPT-3 and Codex solvers that
are capable of solving puzzles -- even without access to any reference
solutions -- by learning from their own past solutions. Codex performs best,
solving up to 18% of 397 test problems with a single try and 80% of the
problems with 1,000 tries per problem. In a small user study, we find a
positive correlation between puzzle-solving performance and coding experience,
and between the puzzle difficulty for humans and AI solvers. Therefore, further
improvements on P3 could have a significant impact on many program synthesis
areas.

23
The categories of this paper are: solv-int
37
The categories of this paper are: supr-con
5
The categories of this paper are: gr-qc
25
The categories of this paper are: patt-sol
8
The categories of this paper are: hep-ex
24
The categories of this paper are: dg-ga
10
The categories of this paper are: q-bio
21
The categories of this paper are: adap-org
32
The categories of this paper are: atom-ph
20
The categories of this paper are: acc-phys
34
The categories of this paper are: chem-ph
14
The categories of this paper are: math-ph
33
The categories of this paper are: plasm-ph
15
The categories of this paper are: nucl-ex
18
The categories of this paper are: econ
11
The categories of this paper are: hep-lat