Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Less is More: Generating Grounded Navigation Instructions from Landmarks   We study the automatic generation of navigation instructions from 360-degree
images captured on indoor routes. Existing generators suffer from poor visual
grounding, causing them to rely on language priors and hallucinate objects. Our
MARKY-MT5 system addresses this by focusing on visual landmarks; it comprises a
first stage landmark detector and a second stage generator -- a multimodal,
multilingual, multitask encoder-decoder. To train it, we bootstrap grounded
landmark annotations on top of the Room-across-Room (RxR) dataset. Using text
parsers, weak supervision from RxR's pose traces, and a multilingual image-text
encoder trained on 1.8b images, we identify 971k English, Hindi and Telugu
landmark descriptions and ground them to specific regions in panoramas. On
Room-to-Room, human wayfinders obtain success rates (SR) of 71% following
MARKY-MT5's instructions, just shy of their 75% SR following human instructions
-- and well above SRs with other generators. Evaluations on RxR's longer,
diverse paths obtain 61-64% SRs on three languages. Generating such
high-quality navigation instructions in novel environments is a step towards
conversational navigation tools and could facilitate larger-scale training of
instruction-following agents.

3
The categories of this paper are: physics
16
The categories of this paper are: stat
10
The categories of this paper are: q-bio
35
The categories of this paper are: mtrl-th
24
The categories of this paper are: dg-ga
6
The categories of this paper are: astro-ph
32
The categories of this paper are: atom-ph
4
The categories of this paper are: cond-mat
14
The categories of this paper are: math-ph
8
The categories of this paper are: hep-ex
17
The categories of this paper are: q-fin
27
The categories of this paper are: comp-gas
13
The categories of this paper are: quant-ph
21
The categories of this paper are: adap-org
18
The categories of this paper are: econ
36
The categories of this paper are: cmp-lg