Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: History Aware Multimodal Transformer for Vision-and-Language Navigation   Vision-and-language navigation (VLN) aims to build autonomous visual agents
that follow instructions and navigate in real scenes. To remember previously
visited locations and actions taken, most approaches to VLN implement memory
using recurrent states. Instead, we introduce a History Aware Multimodal
Transformer (HAMT) to incorporate a long-horizon history into multimodal
decision making. HAMT efficiently encodes all the past panoramic observations
via a hierarchical vision transformer (ViT), which first encodes individual
images with ViT, then models spatial relation between images in a panoramic
observation and finally takes into account temporal relation between panoramas
in the history. It, then, jointly combines text, history and current
observation to predict the next action. We first train HAMT end-to-end using
several proxy tasks including single step action prediction and spatial
relation prediction, and then use reinforcement learning to further improve the
navigation policy. HAMT achieves new state of the art on a broad range of VLN
tasks, including VLN with fine-grained instructions (R2R, RxR), high-level
instructions (R2R-Last, REVERIE), dialogs (CVDN) as well as long-horizon VLN
(R4R, R2R-Back). We demonstrate HAMT to be particularly effective for
navigation tasks with longer trajectories.

14
The categories of this paper are: math-ph
2
The categories of this paper are: math
37
The categories of this paper are: supr-con
24
The categories of this paper are: dg-ga
35
The categories of this paper are: mtrl-th
17
The categories of this paper are: q-fin
20
The categories of this paper are: acc-phys
29
The categories of this paper are: funct-an
13
The categories of this paper are: quant-ph
12
The categories of this paper are: nucl-th
7
The categories of this paper are: hep-th
22
The categories of this paper are: chao-dyn
25
The categories of this paper are: patt-sol
21
The categories of this paper are: adap-org
6
The categories of this paper are: astro-ph
10
The categories of this paper are: q-bio