Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: CoMIR: Contrastive Multimodal Image Representation for Registration   We propose contrastive coding to learn shared, dense image representations,
referred to as CoMIRs (Contrastive Multimodal Image Representations). CoMIRs
enable the registration of multimodal images where existing registration
methods often fail due to a lack of sufficiently similar image structures.
CoMIRs reduce the multimodal registration problem to a monomodal one, in which
general intensity-based, as well as feature-based, registration algorithms can
be applied. The method involves training one neural network per modality on
aligned images, using a contrastive loss based on noise-contrastive estimation
(InfoNCE). Unlike other contrastive coding methods, used for, e.g.,
classification, our approach generates image-like representations that contain
the information shared between modalities. We introduce a novel,
hyperparameter-free modification to InfoNCE, to enforce rotational equivariance
of the learnt representations, a property essential to the registration task.
We assess the extent of achieved rotational equivariance and the stability of
the representations with respect to weight initialization, training set, and
hyperparameter settings, on a remote sensing dataset of RGB and near-infrared
images. We evaluate the learnt representations through registration of a
biomedical dataset of bright-field and second-harmonic generation microscopy
images; two modalities with very little apparent correlation. The proposed
approach based on CoMIRs significantly outperforms registration of
representations created by GAN-based image-to-image translation, as well as a
state-of-the-art, application-specific method which takes additional knowledge
about the data into account. Code is available at:
https://github.com/MIDA-group/CoMIR.

18
The categories of this paper are: econ
13
The categories of this paper are: quant-ph
34
The categories of this paper are: chem-ph
4
The categories of this paper are: cond-mat
8
The categories of this paper are: hep-ex
2
The categories of this paper are: math
16
The categories of this paper are: stat
12
The categories of this paper are: nucl-th
26
The categories of this paper are: bayes-an
20
The categories of this paper are: acc-phys
22
The categories of this paper are: chao-dyn
28
The categories of this paper are: alg-geom
10
The categories of this paper are: q-bio
36
The categories of this paper are: cmp-lg
3
The categories of this paper are: physics
15
The categories of this paper are: nucl-ex