Minimizing _LTR_ essentially maximize the lower
bound of the mutual information _I_ ( _x_ ; _y_ ) (Oord
et al., 2018; Chi et al., 2021). However, it is hard
for models to find an embedding perfectly containing all information of the sentence. Consequently,
models may only pay attention to the high-level
global information and neglect some local tokenlevel information. To this end, we add an auxiliary
loss to force the models to preserve the token-level
information throughout the entire model:
_LRTL_ = [1]
_S_
_S_
_CE_ ( _g_ ( _f∗_ ( _x_ ) _, y_ ) _i, yi_ ) _,_ (6)
_i_ =1
where _f∗_ ( _x_ ) denotes all hidden vectors of _x_ except
CLS and CE denotes cross entropy. It is worth
noting that we do not involve the CLS token in calculating RTL objective because we find it will make
translation ranking objective hard to converge. To
train the RTL head with a stable and consistent
|Direction|xx→en en→xx|Col3|
|---|---|---|
|Model|14 langs
28 langs
36 langs|14 langs
28 langs
36 langs|
|InfoXLM
LaBSE|77.8
-
-
-
-
-|80.6
-
-
-
-
93.7|
|mBERT~~_∗_~~
mBERT (recomputed)
mBERT+TR
mBERT+TR+TLM
**mBERT+DAP**
|-
-
-
42.5
42.2
36.9
94.0
93.8
90.1
94.1
93.8
90.2
**94.7**
**94.7**
**90.9**|45.6
45.1
38.7
43.8
43.3
37.2
93.2
93.4
90.1
93.5
93.5
90.3
**94.2**
**94.6**
**91.2**|
|XLM-R~~_∗_~~
XLM-R (recomputed)
XLM-R+TR
XLM-R+TR+TLM
**XLM-R+DAP**|-
-
-
59.4
60.1
55.3
93.8
94.2
**91.6**
93.2
92.8
89.2
**95.0**
**94.7**
91.3|60.6
63.7
57.7
57.5
58.9
53.3
91.2
91.2
86.4
94.4
94.5
92.4
**95.1**
**95.2**
**92.7**|
Table 1: Average accuracy on Tatoeba bitext retrieval task. Direction "xx _→_ en" means retrieval is performed over
the English corpora, and vice versa. 14 langs and 28 langs mean different subsets of all 36 languages. For mBERT
and XLM-R models, we report both the best implementation before (Results with * are taken from (Hu et al.,
2020)) and our recomputed accuracy. Results of InfoXLM and LaBSE are taken from their papers. For LaBSE we
take the result using mBERT vocabulary for fair comparison. Bold font means that model performs the best among
its group. We use underline to identify a state-of-the-art method that outperforms all our variants.

3.3.2 Representation Translation Learning


To demonstrate the effectiveness of our proposed
Representation Translation Learning, we first compare it with the base models (mBERT or XLM-R)
and their TR-finetuned versions. Additionally, we
also introduce a variant of our method that leverages TLM.
Furthermore, we also compare our approach
with two state-of-the-art multilingual language
models, InfoXLM (Chi et al., 2021) and
LaBSE (Feng et al., 2022). It is worth noting that
InfoXLM and LaBSE use 10 times more training
data than our method and are trained longer with a
larger batch size.

Dual-Alignment Pre-training for Cross-lingual Sentence Embedding

Ziheng Li
Shaohan Huang
Zihan Zhang
Zhi-Hong Deng
Qiang Lou
Haizhen Huang
Jian Jiao
Furu Wei
Weiwei Deng
Qi Zhang


In bitext retrieval, given a query sentence from
source language, models need to retrieve the
most relevant sentence among a collection of sentences in the target language. Following previous
works (Feng et al., 2022; Chi et al., 2021; Artetxe
and Schwenk, 2019b), we use the Tatoeba dataset
to evaluate our pre-training framework in a zeroshot manner.
Tatoeba contains parallel sentences in more than
300 languages, and we use the 36 languages version from XTREME benchmark (Hu et al., 2020).
Each language has up to 1000 sentences paired
with English.
**Results** We test on all 36 languages and report
the average accuracy over 14 languages tested in
LASER (Artetxe and Schwenk, 2019b) and 36 languages tested in XTREME. Besides, we set up a
new group of 28 languages based on our observation of the low-resource test languages. Among
the original 36 languages, some scarce languages
have less than 1000 sentence pairs, and some of
them even only have about 200 sentence pairs, and
we observe that the accuracy of these languages is
inconsistent between the two retrieval directions
("en _→_ xx" and "xx _→_ en" with a difference more
than 30%) and also significantly lower than other
languages with abundant resources. This indicates
that the results obtained from small test sets are
not as reliable as those from larger test sets. Therefore, we report a 28-language version where all
languages contain 1000 test pairs. The retrieval
accuracy for each language is reported in the appendix A.
In Table 1, we observe that our DAP method outperforms all other variants significantly. mBERT
and XLM-R perform the worst because they lack a
sentence-level objective. TLM improves TR’s performance in the direction "en _→_ xx" but hurts direction "xx _→_ en". By contrast, DAP brings consistent
improvement. Compared with the two state-of-theart methods, our method performs much better than
InfoXLM and only slightly falls behind LaBSE.
|Model|fr-en
P R F|de-en
P R F|ru-en
P R F|zh-en
P R F|Avg
F|
|---|---|---|---|---|---|
|LaBSE|96.3 93.6 95.0|99.4 95.4 97.3|99.3 93.1 96.1|90.4 88.3 89.4|94.5|
|mBERT (recomputed)
mBERT+TR
mBERT+TR+TLM
**mBERT+DAP**|75.1 68.2 71.5
96.1 90.9 93.4
95.6 90.9 93.2
95.1 94.1 94.6|77.8 69.0 73.1
  98.8 94.0 96.3
  98.3 94.0 96.1
  98.1 94.7 96.4|70.1 52.9 60.3
   98.4 89.8 93.9
   97.0 89.7 93.2
   98.6 91.4 94.9|63.1 50.6 56.2
     96.0 93.8 94.9
     93.9 95.7 94.8
     95.7 94.2 94.9|65.3
      94.6
      94.3
** 95.2**|
|XLM-R (recomputed)
XLM-R+TR
XLM-R+TR+TLM
**XLM-R+DAP**|81.3 68.2 74.2
92.6 92.1 92.4
91.4 91.6 91.5
95.3 93.1 94.2|86.6 77.0 81.5
  96.3 94.6 95.4
  94.0 95.5 94.7
  99.0 95.2 97.1|87.6 74.0 80.2
   97.3 91.0 94.0
   94.4 90.9 92.7
   98.1 93.3 95.6|77.0 54.9 64.1
     96.6 87.5 91.8
     92.8 90.3 91.5
     96.7 92.6 94.6|75.0
      93.4
      92.6
** 95.4**|
Table 2: Evaluation on BUCC training set. The thresholds are chosen to achieve the optimal F1 score.
|Model|fr-en
P R F|de-en
P R F|ru-en
P R F|zh-en
P R F|Avg
F|
|---|---|---|---|---|---|
|LaBSE
|92.8 82.5 87.4|96.6 85.2 90.5|91.2 85.9 88.5|85.5 70.4 77.2|85.9|
|mBERT~~_∗_~~
mBERT (recomputed)
mBERT+TR
mBERT+TR+TLM
**mBERT+DAP**
|-
-
62.6
 80.1 42.1 55.2
93.6 75.2 83.4
92.4 75.0 82.8
92.1 83.4 87.6|-
-
62.5
   83.7 38.2 52.5
  97.3 77.1 86.0
  96.2 78.2 86.3
  96.2 83.6 89.5|-
-
51.8
    69.1 28.9 40.8
   91.3 77.2 83.6
   90.1 77.2 83.1
   90.1 82.4 86.1|-
-
50.0
      65.8 20.2 30.9
     93.0 69.7 79.7
     90.9 75.8 82.6
     92.5 75.7 83.3|56.7
       44.8
      83.2
      83.7
** 86.6**|
|XLM-R~~_∗_~~
XLM-R (recomputed)
XLM-R+TR
XLM-R+TR+TLM
**XLM-R+DAP**|-
-
67.5
 85.9 47.3 61.0
89.7 79.1 84.1
88.1 75.8 81.5
92.1 82.1 86.8|-
-
66.5
   88.6 48.3 62.5
  94.2 80.3 86.7
  91.2 79.8 85.1
  96.6 81.1 88.2|-
-
73.5
    85.8 54.3 66.5
   89.6 80.2 84.7
   86.3 80.6 83.4
   89.5 88.1 88.8|-
-
56.7
      77.7 27.3 40.4
     92.2 66.1 77.0
     89.6 72.6 80.2
     93.7 75.0 83.3|66.0
       57.6
      83.1
      82.5
** 86.8**|
Table 3: Evaluation on BUCC test set. The thresholds are chosen to achieve the optimal F1 score on training
set. For mBERT and XLM-R models, we report both the best implementation before (Results with * are taken
from (Hu et al., 2020)) and our recomputed scores.
Considering the training cost, we think this result
has demonstrated DAP’s potential.


In bitext mining, models need to detect the parallel sentence pairs (e.g., translations) from a pair
of monolingual corpus. We use the BUCC 2018
dataset (Zweigenbaum et al., 2017) to perform evaluations, which contains four language pairs: fr-en,
de-en, ru-en and zh-en. Each corpus contains 150k
to 1.2M unpaired sentences and gold labels telling
which sentences are translation pairs.
Following Artetxe and Schwenk (2019a), we
employ the ratio between the cosine of a given candidate and the average cosine of its neighbours
in both directions. The training set is used to
learn the best threshold (Schwenk, 2018) to decide which pairs should be selected. More details
of the scoring function and threshold can be found
in appendix B.
**Results** Table 2 shows the precision, recall and
F1 score for four language pairs on training set after
optimization. The results of LaBSE are produced
using the checkpoints publicized in Huggingface
model hub. We do not report the results of InfoXLM because this task was not evaluated in the
original paper and we failed to produce reasonable
results.
Our method outperforms all variants and even
LaBSE, which means our model learns an embedding space with better separability. When testing
the optimized model on test set, our model shows
remarkable generalization ability and enlarges the
gap against other methods as shown in Table 3.
We outperform the state-of-the-art LaBSE by 0.9%
and other variants by at least 3.0%. Similar to the
retrieval task, mBERT and XLM-R perform the
|Model|en fr es de el bg ru tr ar vi th zh hi sw ur Avg|
|---|---|
|InfoXLM
LaBSE|86.4 80.3 80.9 79.3 77.8 79.3 77.6 75.6 74.2 77.1 74.6 77.0 72.2 67.5 67.3 76.5
85.4 80.2 80.5 78.8 78.6 80.1 77.5 75.1 75.0 76.5 69.0 75.8 71.9 71.5 68.1 76.3|
|mBERT
mBERT+TR
mBERT+TR+TLM
**mBERT+DAP**|82.1 74.4 74.9 71.2 67.9 69.5 69.6 62.8 66.2 70.6 54.6 69.7 60.4 50.9 58.0 66.8
82.0 74.3 75.1 72.9 69.9 73.1 70.6 68.6 67.4 73.6 61.3 70.8 65.0 62.6 61.0 69.9
 82.8 75.2 74.4 72.0 69.3 70.6 69.4 66.1 66.1 70.6 58.9 67.3 63.7 60.6 59.5 68.4
81.8 75.6 76.2 74.4 72.6 74.9 72.0 71.3 69.7 74.4 63.6 72.3 67.3 67.3 63.2** 71.8**|
|XLM-R
XLM-R+TR
XLM-R+TR+TLM
**XLM-R+DAP**|83.8 77.6 78.2 75.4 75.0 77.0 74.8 72.7 72.0 74.5 72.1 72.9 69.6 64.2 66.0 73.7
83.5 76.4 76.8 75.7 74.2 76.2 74.6 71.8 71.1 74.2 69.1 72.9 68.8 66.8 65.2 73.1
 84.6 77.4 76.9 74.9 68.1 69.8 69.4 68.1 61.7 68.9 62.6 66.9 61.4 61.7 57.5 68.7
82.9 77.0 77.7 75.7 75.2 76.0 74.7 73.1 72.5 74.2 71.9 73.0 69.8 70.5 66.0** 74.0**|
Table 4: Accuracy for XNLI cross-lingual natural language inference. Results of InfoXLM are taken from their
paper.
worst. TLM brings improvements for zh-en but
gets worse for fr-en. DAP consistently performs
the best on all metrics. Furthermore, the improvement observed in DAP’s performance is larger in
comparison to the retrieval task. This indicates that
DAP is more effective in enhancing performance
on complex tasks, suggesting its potential as a valuable tool for addressing challenging problems.


Natural language inference (NLI) is a well-known
task to evaluate models’ classification performance
under fine-tuning. The goal is to predict the relationship between the input sentence pair. The candidate relationships are entailment, contradiction and
neutral. XNLI (Conneau et al., 2018) extends NLI
to the multilingual setting of 15 languages. Following Chi et al. (2021), we fine-tune the model with
the English training set and directly evaluate on
test sets of other languages. The hyperparameters
of fine-tuning are reported in the appendix C.
**Results** Table 4 shows accuracy for 15 languages.
We observe that the differences between variants
are relatively small compared with retrieval and
mining tasks. We think this is because judging
the relationship between two sentences does not
rely on cosine similarity, so the pre-training cannot be directly transferred to the downstream task.
mBERT variants all show positive results and DAP
has the largest improvement. But for XLM-R variants, only DAP maintains the performance as the
base model. The TR and TLM variants suffer
from performance degradation. We think this is
because XLM-R has already been a well-trained
|Direction|Tatoeba BUCC XNLI|
|---|---|
|xx_→_en
en_→_xx
Both|**91.0**
**86.6**
**71.8**
90.5
84.1
69.3
90.8
86.3
70.5|
Table 5: Performance of different RTL directions
across three tasks. "xx _→_ en" means RTL head reconstructs English sentences using non-English token representations, and vice versa. "Both" means we calculate the RTL loss from both directions on half of the
batch respectively and take the average.
multilingual model and our continued pre-training
is insufficient to improve the classification capacity. However, we demonstrate DAP will not harm
classification performance for a well-trained base
model.


In this section, we conduct experiments to get a
deeper understanding of DAP. In each setting, we
report the average accuracy over 36 languages and
two retrieval directions on Tatoeba, average F1
score on BUCC test set and average accuracy on
XNLI. All variants are trained from mBERT.


In our method, the RTL head only learns to translate from non-English to English. Here we investigate if the opposite direction can help the pretraining. To remind the model of the language to
be reconstructed, we add language embeddings to
the representation before the RTL head like TLM.
As shown in Table 5, translating from English to
non-English performs much worse than the opposite direction. Also, the mixed-up training gets an
Figure 3: Performance of varying reconstruction ratios
across three tasks.
intermediate performance. We attribute the difference between the two directions to the dispersion
of the objective. We assume that RTL aligns the
source language’s representation towards the target
language. So, if the reconstruction target keeps
switching among different languages, it will make
RTL hard to converge.


To better understand the objective of the RTL task,
we conduct experiments where RTL head only
needs to reconstruct partial target sentences with
the other target token representations accessible.
The tokens to reconstruct are selected randomly
with probability _ρ_ . Larger _ρ_ will make the RTL
task harder.
From Figure 3, we can find the variants with _ρ <_
1 have similar performance on all tasks and there is
a steep increase at _ρ_ = 1. We think this is because
the unmasked target token representations cause
information leakage, so the RTL head does not
need to learn the alignment from source sentences.


We investigate the relation between the RTL head’s
complexity and the pre-training performance. We
set _K_ = 1 _,_ 2 _,_ 3 _,_ 4 to give RTL head different capabilities to extract aligned information from the
representation of the source sentence.
In Figure 4, the three tasks show different tendencies with regard to RTL head’s complexity. Only
the accuracy on Tatoeba keeps increasing along
with _K_ but the gain from larger _K_ is declining
especially after _K_ = 2. For the other two tasks,
larger _K_ brings a negative effect. We hypothe
Figure 4: Performance of varying numbers of RTL
head layers across three tasks.
|Model|FLOPs Latency|
|---|---|
|mBERT+TR
mBERT+TR+TLM
**mBERT+DAP**|11.0G
0.51
33.7G
1.34
16.5G
0.88|
Table 6: Computational efficiency of different pretraining methods. The unit of latency is milliseconds
per sample.
size that a smaller _K_ that makes RTL task harder
will enforce the model to generate more informative representations. Setting _K_ = 2 achieves the
best general cross-lingual performance across three
tasks.


Computational efficiency is an important factor
when designing pre-training tasks. A more efficient
method enables models to train on a larger dataset
for more steps. We calculate the feedforward floating point operations (FLOPs) for our method and
TLM, respectively. In addition, we report the training latency in our training environment. We measure the latency with a total batch size of 512 on 8
Tesla V100 GPUs using PyTorch distributed data
parallel.
From Table 6, we can find DAP only increases
the training cost by about 50% against the TR-only
baseline, which can be further improved if we use
negative sampling to reduce the softmax over the
huge vocabulary. By contrast, TLM introduces
a training cost of more than 150% due to the extra feedforward propagation through the 12-layer
encoder. Therefore, DAP is more efficient and scalable for cross-lingual pre-training.


In this paper, we find that token-level alignment is
crucial for cross-lingual tasks. Based on this observation, we present a dual-alignment pre-training
framework for cross-lingual sentence embedding
that enables both sentence-level and token-level
alignment. The framework consists of a translation
ranking task and a newly proposed representation
translation learning task, which encourages the token representation to contain all information from
its translation counterpart in an efficient way.
We train our models on a moderate-size corpus. The model trained with DAP significantly
outperforms variants without token-level alignment
or using TLM as the alignment task across three
sentence-level cross-lingual tasks, and achieves performance comparable with those state-of-the-art
pre-training work trained on 10 times more data
with larger batch size and training steps. These
results show our approach brings essential improvement for cross-lingual sentence embedding.


Although our method is efficient and scalable, we
have not conducted pre-training on large-scale corpora due to limited computational resources. The
quality and quantity of data are crucial factors for
a pre-training model. As our model only covers
36 languages, it cannot provide services for many
rare languages. This paper just proposes a new pretraining direction and does not use many training
tricks. Exploring DAP’s full capability is left for
future work.
Besides, RTL task is not the only possible tokenalignment task for our DAP framework. Other
objectives based on token representations are also
worth investigating. The best objective form is still
under research.


We report the Tatoeba retrieval accuracy of all 36
languages in Table 7 and Table 8. Our approach
consistently outperforms other baselines in both
directions for most languages, with the advantage
being particularly significant in the "en _→_ xx" direction. We observed that the performance of the
TR-only model can vary much between the two
directions, as demonstrated by languages such as
jv, kk, sw, and tl. In contrast, our approach exhibits
much more stable performance, which is beneficial
for bidirectional applications.


In contrast to direction comparison between similarities, margin-based method accounts for the
scale inconsistencies of measure. We adopted the
method proposed by Artetxe and Schwenk (2019a):
(8)
where _Nk_ ( _x_ ) denotes the set of _k_ nearest neighbours of _x_ in the other language. In our experiments, we set _k_ = 4.
With a certain threshold _γ_, sentence pairs such
that _f_ ( _x, y_ ) _≥_ _γ_ are identified as aligned. For those
_x_ appearing in multiple aligned pairs, we select the
pair with the highest score.
To decide the best threshold, we first compute
the scores of all candidates and sort them into an
ordered sequence. Next, we compute F1 score by
_φ_ ( _x, y_ )
_f_ ( _x, y_ ) =
~~�~~ _z∈Nk_ ( _x_ ) _φ_ ( _x,yk_ ) +
_,_
_z,y_ )
_k_
_x,y_ ) _φ_ ( _z,y_ )
_k_ + ~~[�]~~ _z∈Nk_ ( _y_ ) _k_
|Model|af ar bg bn de el es et eu fa fi fr he hi hu id it ja|
|---|---|
|mBERT+TR
mBERT+TR+TLM
**mBERT+DAP**|95.5 90.3 94.5 88.8 99.1 96.4 98.1 97.4 95.1 94.0 96.5 95.3 90.6 95.5 96.8 95.4 94.4 96.1
 95.9 89.7 94.6 87.0 99.1 95.6 98.3 96.8 95.0 94.0 95.7 95.4 91.5 95.6 95.6 95.0 93.8 95.0
96.9 91.8 95.4 89.3 99.1 96.8 98.4 98.0 96.2 95.9 97.1 95.5 93.0 96.8 97.0 95.9 95.5 96.7|
|XLM-R+TR
XLM-R+TR+TLM
**XLM-R+DAP**|95.0 90.0 92.9 89.3 99.1 93.9 98.1 97.8 95.3 95.3 96.9 95.3 91.1 96.4 97.0 95.1 94.4 96.1
 92.7 90.2 94.3 88.8 99.1 95.5 97.3 96.8 93.8 94.4 95.9 94.2 91.2 96.4 95.9 96.0 94.4 94.2
96.1 93.1 95.7 91.4 99.2 96.7 98.4 98.1 96.0 94.9 97.3 95.5 93.6 97.3 97.0 96.4 96.3 96.2|
||jv
ka
kk
ko
ml
mr
nl
pt
ru
sw
ta
te
th
tl
tr
ur
vi
zh|
|mBERT+TR
mBERT+TR+TLM
**mBERT+DAP**|29.3 81.0 62.6 91.2 97.7 91.6 96.2 95.4 95.6 75.1 84.0 90.2 96.2 67.7 98.2 89.6 96.9 95.3
 31.2 79.2 64.7 91.8 97.5 92.0 95.9 95.4 94.8 77.2 85.3 89.7 96.0 71.0 97.7 91.3 96.9 95.3
30.2 79.9 63.8 93.2 98.5 92.5 96.6 96.2 95.5 77.9 83.1 88.5 96.9 70.1 98.5 90.8 97.5 95.4|
|XLM-R+TR
XLM-R+TR+TLM
**XLM-R+DAP**|46.3 90.5 75.7 92.7 98.5 93.2 96.7 95.4 94.7 73.3 84.4 93.6 96.7 74.2 97.2 91.6 97.5 95.7
 23.4 92.4 69.2 91.6 97.2 90.4 95.7 95.5 94.3 72.8 71.0 88.5 96.4 55.8 97.1 85.9 97.0 94.6
27.3 93.7 68.5 93.3 98.4 92.5 96.6 96.1 95.4 77.2 80.8 92.3 98.2 65.6 98.3 90.3 98.2 95.4|
Table 7: Retrieval accuracy on 36 languages of direction xx _→_ en.
|Model|af ar bg bn de el es et eu fa fi fr he hi hu id it ja|
|---|---|
|mBERT+TR
mBERT+TR+TLM
**mBERT+DAP**|94.8 88.7 93.3 86.2 98.8 95.4 97.4 96.3 94.7 94.3 95.6 95.8 89.7 95.0 95.6 94.3 95.1 95.9
 95.7 88.0 93.8 85.8 98.9 96.1 97.6 96.3 94.8 93.7 94.8 95.3 89.6 95.3 94.4 94.1 94.1 95.3
96.3 90.6 94.3 87.8 98.9 96.1 98.1 98.0 96.0 95.6 96.4 95.4 92.2 96.0 96.5 95.2 95.8 96.6|
|XLM-R+TR
XLM-R+TR+TLM
**XLM-R+DAP**|87.6 90.3 92.0 85.5 98.3 95.9 96.2 95.9 92.8 93.1 95.4 92.4 91.6 94.3 95.6 94.0 94.4 90.9
 96.1 89.3 93.9 90.0 99.1 93.9 98.2 97.0 94.9 95.7 96.8 95.4 89.6 97.1 96.5 95.3 94.4 96.4
96.3 92.2 95.4 91.2 98.9 96.6 98.6 98.1 95.7 96.0 97.1 96.3 93.1 97.0 97.2 96.3 96.1 97.3|
||jv
ka
kk
ko
ml
mr
nl
pt
ru
sw
ta
te
th
tl
tr
ur
vi
zh|
|mBERT+TR
mBERT+TR+TLM
**mBERT+DAP**|43.4 81.5 66.4 91.8 97.4 92.3 96.1 94.6 94.8 72.3 83.4 89.3 95.8 70.6 96.8 89.5 97.3 94.3
 46.3 78.0 67.8 92.5 98.0 92.2 95.9 94.7 94.2 74.9 84.0 89.7 95.8 74.6 96.8 90.4 97.6 94.9
47.3 80.8 65.4 92.3 98.3 93.3 97.2 95.6 94.8 75.6 82.4 89.7 96.4 75.5 98.2 91.7 97.8 95.3|
|XLM-R+TR
XLM-R+TR+TLM
**XLM-R+DAP**|16.1 88.3 57.6 89.8 96.2 87.3 95.4 95.5 93.9 59.5 62.5 81.6 95.3 46.8 97.0 82.2 96.7 92.8
 49.8 90.6 82.6 92.4 98.5 94.2 97.0 95.0 94.2 81.5 86.0 96.6 96.9 80.2 96.6 92.6 97.7 95.2
47.3 91.6 75.3 93.4 99.0 93.6 96.8 95.6 95.1 78.5 86.3 94.9 97.8 77.1 97.9 92.7 98.0 96.0|
Table 8: Retrieval accuracy on 36 languages of direction en _→_ xx.
setting _γ_ to each middle point of two consecutive
scores and find the optimal _γ_ . This procedure is
done on training set.


The fine-tuning hyperparamter setting is shown in
Table 9. We searched the learning rate among {1e5, 3e-5, 5e-5, 7e-5}.
Batch size 256
Learning rate 5e-5
Epochs 2
Max seq length 128
Weight decay 0
Table 9: Hyperparameter setting of XNLI fine-tuning.


Steven Tel [1] _[,]_ [5][*] Zongwei Wu [2] _[,]_ [5][*] Yulun Zhang [3][ †] Barth´el´emy Heyrman [1]
C´edric Demonceaux [4] _[,]_ [5] Radu Timofte [2] Dominique Ginhac [5]
1 University of Burgundy, ImViA 2 Computer Vision Lab, CAIDAS & IFI, University of W¨urzburg
3 CVL, ETH Z¨urich 4 University of Lorraine, CNRS, Inria, Loria 5 University of Burgundy, CNRS, ICB
_{_ steven.tel; barthelemy.heyrman; cedric.demonceaux; dginhac _}_ @u-bourgogne.fr, _{_ zongwei.wu; radu.timofte _}_ @uni-wuerzburg.de, yulun100@gmail.com

Alignment-free HDR Deghosting with Semantics Consistent Transformer

Steven Tel
Zongwei Wu
Yulun Zhang
Barthélémy Heyrman
Cédric Demonceaux
Radu Timofte
Dominique Ginhac


_High dynamic range (HDR) imaging aims to retrieve in-_
_formation from multiple low-dynamic range inputs to gen-_
_erate realistic output. The essence is to leverage the con-_
_textual information, including both dynamic and static se-_
_mantics, for better image generation. Existing methods of-_
_ten focus on the spatial misalignment across input frames_
_caused by the foreground and/or camera motion. However,_
_there is no research on jointly leveraging the dynamic and_
_static context in a simultaneous manner. To delve into this_
_problem, we propose a novel alignment-free network with a_
_Semantics Consistent Transformer (SCTNet) with both spa-_
_tial and channel attention modules in the network. The spa-_
_tial attention aims to deal with the intra-image correlation_
_to model the dynamic motion, while the channel attention_
_enables the inter-image intertwining to enhance the seman-_
_tic consistency across frames. Aside from this, we intro-_
_duce a novel realistic HDR dataset with more variations in_
_foreground objects, environmental factors, and larger mo-_
_tions. Extensive comparisons on both conventional datasets_
_and ours validate the effectiveness of our method, achiev-_
_ing the best trade-off on the performance and the compu-_
_tational cost. The source code and dataset are available at_
_[https://steven-tel.github.io/sctnet/.](https://steven-tel.github.io/sctnet/)_


While dealing with the full range of illumination in natural scenes, traditional cameras fail to provide images with
reasonable quality and result in the loss of information in
under-exposed and over-exposed areas. Some specialized
hardware devices [18, 29] have been proposed to produce
HDR images directly. Unfortunately, they are usually too
expensive to be widely adopted. To this end, computational
High Dynamic Range (HDR) imaging has recently attracted
*Both authors contributed equally to this research.
†Corresponding Author: Yulun Zhang
Figure 1. Motivation of our alignment-free network. Most existing
methods [15, 27, 8, 24, 34] calibrate the LDR inputs at shallow
latent space vis-`a-vis the dynamic motion. Differently, we propose
an alignment-free network that jointly models the dynamic and
static context across frames through attention blocks.
great research attention, aiming to merge multiple Low Dynamic Range (LDR) images with different exposure times
to produce an HDR image with increased dynamic range
and heightened realism.
Given well-aligned LDR images, traditional imaging
methods [4, 25, 7, 10, 31] can produce faithful results. Nevertheless, a perfect alignment is always hard to achieve.
In practice, it is very common to have dynamic objects in
the foreground and/or with ego motion, resulting in undesirable misalignment between the LDR inputs. Moreover,
the movement across frames with different exposure values (EV) may lead to missing content in the over-/underexposed regions. Therefore, a method that can efficiently
leverage information from multiple exposures, while being
robust to the misalignment is highly desired.
To tackle the misalignment issue due to the moving object, multiple methods [2, 11] propose to compute the optical flow to model the foreground motion. Several recent
methods [10, 17, 3] further integrate optical flow into a
learning-based method to align motion between the input
LDR Inputs
Prediction
Concatenation
images. However, due to the poor generalization of optical flow methods on multiple exposures, these methods fail
to produce reliable results. Other methods [32, 15] compute the affinity matrices between different images with the
help of attention modules as shown in Figure 1. Once the
multi-EV features are calibrated, they are further merged
and fed into deep convolutional neural networks (CNNs)
such as [36]. More recently, Niu _et al_ . [19] leverages generative adversarial network (GAN) for HDR imaging to deal
with ghost artifacts. [8] further merges GAN with an attention module to improve the model efficiency. However,
these methods are based on local convolutions, limiting
their ability to model and benefit from long-range dependencies. A recent method [15] replaces the deep convolutional network with a transformer and proposes a hybrid
CNN-transformer architecture. In general, existing methods
[15, 32, 8, 24, 34]often follow the common practice of first
computing the cross-image alignment with convolutional
attention. Then, the features are further refined with contextualized awareness. Despite the plausible performance,
these methods are heavily based on the alignment module
for multi-EV feature calibration. Moreover, the SOTA performance is achieved with heavy computational cost, which
is undesirable for real-time applications.
We observe that previous works [24, 34, 3, 32, 27] often
focus on dealing with the motion changes across different
images. However, motion changes are highly correlated, not
decoupled, with the static contexts, _i.e_ ., the semantic information of the holistic environment during imaging. Inspired
by this observation, in this paper, we propose SCTNet, a
novel Semantics Consistent Transformer network, which
consists of two parts: intra-image spatial feature modeling
and inter-EV semantic intertwining. Specifically, instead
of computing the attention at a shallow stage, as shown in
Figure 1, we directly investigate the transformer network to
explore the spatial correlation between the merged multiEV inputs. This spatial feature modeling operation aims
to not only compute the contextualized awareness within a
single image but also explore the spatial correlation across
LDR inputs. However, directly aggregating the multi-input
features is challenging since the different exposures may
have different spatial distributions. Thus, we complement
the spatial attention block with our new semantic-consistent
cross-frame attention block. These two attention blocks
work in a consecutive manner to make sure that both the
foreground motion and the static context are well-learned
and merged simultaneously in an end-to-end manner.
Moreover, to fully benefit from the popular non-local attention mechanism, we introduce a novel dataset for HDR
deghosting that adheres to the standard acquisition approach put forth by [10]. Our dataset is collected to fulfill the demand for larger training and evaluation data required to implement vision transformers in HDR deghost
ing tasks. The dataset comprises diverse global and local
motion and illumination ranges to generate more versatile
and all-encompassing HDR deghosting solutions. To make
algorithms more robust, we have expanded our dataset to include a wider range of scenes, including twilight and night
scenes, in contrast to the conventional dataset [10] with only
day scenes. To summarize, our contributions are threefold:
  - We propose a unified framework, SCTNet, that jointly
models the foreground motion and static semantic context simultaneously, both of which are essential for
HDR deghosting.
  - We introduce a realistic supervised HDR deghosting
dataset presenting a richer variety of scenes with more
variation in the types of motion and more environmental factors than existing benchmark datasets.
  - We demonstrate the effectiveness of our approach
through extensive experiments on both the Kalantari
dataset [10] as well as our proposed dataset. Our SCTNet performs favorably against other state-of-the-art
techniques by reaching the best trade-off between performance and computational cost.


**HDR Deghosting Methods:** Existing HDR deghosting algorithms can be categorized into two groups: Patch MatchBased and Deep Learning methods.
_**Patch Match-Based Methods**_ : One category involves pixel
rejection techniques. These methods focus on globally
aligning LDR images and then discarding misaligned pixels. Approaches include error map creation based on color
differences by Grosch _et al_ . [6], motion area identification
using threshold bitmaps by Pece _et al_ . [20], and weighted
intensity variance analysis by Jacobs _et al_ . [9]. Other methods propose gradient-domain weight maps by Zhang _et_
_al_ . [35] and probability maps by Khan _et al_ . [12]. However, such methods often lead to suboptimal HDR results
due to pixel rejection causing information loss.
Another set of methods focuses on motion registration.
These involve aligning non-reference LDR images with the
reference LDR image prior to merging. Techniques include optical flow-based motion vector prediction by Bogoni [2], luminance domain transformation using exposure time and optical flow-based motion compensation by
Kang _et al_ . [11], and optical flow-based registration followed by HDR reconstruction by Zimmer _et al_ . [37]. Additionally, Sen _et al_ . [25] introduced a patch-based energy
minimization for simultaneous alignment and HDR reconstruction. Hu _et al_ . [7] enhanced image alignment through
brightness and gradient consistency optimization.
_**Deep Learning Solutions**_ : To address occlusions in patchbased methods, convolutional neural networks (CNNs)
gained prominence due to their ability to handle such
challenges more effectively. Kalantari _et al_ . [10] pioneered the integration of a sequential neural network into
the alignment-before-merging pipeline using optical flow.
Wu _et al_ . [31] adopted a direct domain translation network
for ghost-free HDR generation. Further solutions utilized
attention mechanisms to better handle long-range ghosting
effects. Yan _et al_ . [32] employed spatial attention, while
Yan _et al_ . [33] used a non-local network for merging. Niu _et_
_al_ . [19] introduced a GAN-based training scheme for highquality ghost-free HDR generation.
Transformers have emerged as a promising solution due
to their self-attention mechanism. Song _et al_ . [26] proposed
a transformer-based selective HDR reconstruction network,
and Liu _et al_ . [15] introduced a hierarchical network combining Transformers and convolutional channel attention to
capture global and local dependencies, respectively.
**HDR Deghosting Datasets:** The first dataset with HDR labels for training supervised HDR deghosting networks have
been proposed by Kalantari _et al_ . [10]. Each scene is composed of a set of three LDR input images of the same scene
separated by two or three exposure values (EV). Motion in
each input image is achieved using a human subject moving
in the foreground of a static scene. For each scene, the corresponding HDR ground truth is generated from a static set
of three input images (in which the subject is requested to
stay still during the acquisitions), using a triangle weighting function similar to that proposed by Debevec _et al_ . [4].
The middle exposure input from the static bracket is used
as the middle input of the sample. This method enables the
creation of realistic HDR ground truth. However, acquiring a large number of samples can be a tedious and timeconsuming process, as it involves multiple iterations to ensure that scenes without any motion are selected for ground
truth generation. For example, Prahakbar _et al_ . [23] utilized
this method to create a dataset. Nonetheless, the resulting
ground truth images often contain ghosting artifacts.
Another approach for generating a three-input HDR
deghosting dataset was introduced by the NTIRE HDR
challenge [21]. To create a more diverse dataset, the
HDR ground-truth images were obtained from the work of
Froelich _et al_ . [5], who captured a comprehensive collection of HDR videos using a professional two-camera rig
with a semitransparent mirror for the purpose of evaluating HDR displays. Since these images do not have the required LDR input images, the corresponding LDR counterparts were generated synthetically using image formation
models that include noise sources.


In Table 2, we highlight the strengths of our dataset over
existing HDR datasets. Specifically, compared to the widely
Table 1. Comparison against existing benchmarks. Our proposed
benchmark is more complete, providing more realistic and challenging samples for HDR deghosting.
Dataset P.A R.W U.G.T V.L.C. I.C.
Kalantari _et al_ . [10] ✓ ✓ ✓ ✗ ✓
Prabhakar _et al_ . [23] ✗ ✓ ✗ ✗ ✓
NTIRE21 [21] ✓ ✗ ✓ ✓ ✗
**Ours** ✓ ✓ ✓ ✓ ✓
Attribute Description:
P.A. Publicly Accessible
R.W. Real World dataset
U.G.T. Unghosted Ground Truth
V.L.C. Variety of Light Conditions
I.C. Information Consistency
Night
Sunset
Daylight
Local motion Ego motion Full motion
Figure 2. Scene and motion distributions of our proposed dataset.
used Kalantari [10] dataset, our dataset offers a wider variety of scenes including day, twilight, and night scenarios,
while Kalantari only includes scenes captured during the
day. Moreover, the variety of movements in our samples is
richer because our dataset includes not only the human subjects as foreground motion but also ego motion and combined human subjects/camera motion. Besides, our dataset
contains more scene samples for both training and testing,
_i.e_ ., 108/36 training/testing samples compared to 74/15 in
the Kalantari dataset. It is also noteworthy that the scene
variation of the Kalantary dataset is limited to the daytime and only 3 among the 15 testing samples contain extremely light camera motion, restricting its evaluative capacity. Conversely, our dataset addresses these limitations
with a more **balanced scene and motion distribution**, as
shown in Figure 2. We collected samples from diverse light
conditions (day, sunset, and night) and encompassed **richer**
**motion types** (local, ego, and full motions) in equal proportions, for both train/test samples. This diverse motion
range facilitates a comprehensive assessment of HDR methods across challenging scenes.
During recent challenges [21], other HDR samples have
been released. However, the associated LDR images are
generated from HDR images with different sources of synthetic noise, resulting in non-faithful inputs. As shown in
Figure 3, there may exist information in the ground truth
(GT) images, while being not retrievable from the input
LDR images. Finally, the recent Prabhakar dataset [23]
claims to have more than 500 samples, but, until now, only
32 of them are publicly available. Moreover, as shown in
Figure 3, the GT images from the Prabhakar dataset may
contain ghosts, which severely limits its application.
In contrast to these state-of-the-art datasets, our dataset is
more realistically complete because it was built from high
(A) (B)
(C) Samples from our proposed dataset
Figure 3. Comparison with existing datasets. (A) NTIRE dataset [21] may contain content in the ground truth (GT) images while not being
available from the LDR images. (B) Prabhakar [23] dataset generates the GT image with ghosts, which is mainly due to the background
motion in the LDR images. (C) Our dataset contains challenging samples with different kinds of motion. From left to right: local motion
where only people are moving, ego motion where only the camera is moving, and full motion where both foreground and camera are
moving. Please zoom in for more details.
quality LDR inputs and artifact-free HDR ground truth.
Thanks to the large variation in both motion types and lighting conditions, our dataset can be naturally used for analyzing the generalization capability across different scenarios.

3.1. Comparison with state-of-the-art HDR datasets


Our objective is to produce a set of LDR images with
controlled motion and the GT HDR image. Before any acquisition, we first select the exposure time of the reference
image to be certain to get the most information on the scene.
For the acquisition process, we use bracketings of 9 images
spaced by _±_ i-EV difference where _i ∈_ 1 _,_ 2 _,_ 3 _,_ 4 around the
reference exposure. Using 9 exposure values guarantees to
cover the full dynamic range of a scene. A tripod-mounted
camera is used to perform sequentially the 9 acquisitions of
a static scene, for which we check that no parasitic motion
exists. All the images are captured in raw format with a
resolution of 4256 x 2832 pixels using a Nikon D700 camera. Motion in each scene is controlled by a stop motion
technique including 3 stops of 9 images. In order to provide an enhanced dataset including greater opportunities to
accurately assess the deghosting capabilities of current and
future HDR solutions, we propose three different kinds of
motions for each scene, as illustrated in Figure 3:
**Local motion:** The scene is static. Movement between
frames is introduced by people or objects in the foreground;
**Ego motion:** Background and foreground of the scene are
static, the deghosting challenge is introduced in all of the
pixels of the scene due to the camera ego motion;
**Full motion:** Composed of both ego and local motions,
these scenarios are the most challenging.


Since there is no motion between the 9 images of each
stop, the method proposed by Debevec [4] is applied to produce a ground truth HDR image (named _HDR_ 9) for the
second stop by merging the 9 input images. Following the
common HDR practice with only 3 LDR inputs, we also
have to select the 3 best images from the 9 available candidates in each stop, while maintaining the most information
possible. For this purpose, the reference image and the images with _±_ i-EV difference, _i ∈_ 1 _,_ 2 _,_ 3 _,_ 4, are selected to
produce the different _HDR_ 3 _i_ images. In order to preserve
the informative features from each input image, the triangle
functions proposed by [10] are used as blending weights _wn_
to generate the ground truth HDR images:
l
_{_ eq _[:][b]_ [l] _[e]_ [n] _[d][i]_ [n] _[g]_ [f]
_\_ l _a_ b e ~~u~~ n _c_ ti _[o][n]_ [}] [\] _hat {H}(p) = \frac {\sum _{n=1}^{3} w_n(p)H_n(p)}{\sum _{n=1}^{3} w_n(p)},_ (1)
where the resulting HDR image _H_ [ˆ] at the pixel _p_ is the
sum of the projected HDR inputs _Hn_ obtained following
the gamma projection function described in Eq. 2.
By computing the HDR-VDP2 [16] metrics between
_HDR_ 9 and each of the _HDR_ 3 _i_ images, we choose the
_HDR_ 3 _i_ producing the best score as the ground truth HDR
image of the sample scene, leading to 200 samples. To ensure high-quality of our dataset, we invited five viewers for
quality control and rejected the unfaithful results. Our final
benchmark contains 108/36 training/testing samples.


We define _Lk_ as the input sequence of _k_ LDR images
from a dynamic scene with different exposures, HDR imaging aims at reconstructing an HDR image _H_ [ˆ] aligned to a
selected reference image _LR_ . Following the common practice, we chose _k_ = 3 as the number of input images, and
select the middle exposure _L_ 2 input as the reference input.
To increase the robustness of our solution to exposure differences between input frames, we compute the respective
projection of each LDR input frame into the HDR domain
𝑓�
𝑓�
𝑓�;𝑓�; 𝑓�
|Col1|Col2|
|---|---|
|C-MCA|C-MCA|
|P|Col2|
|---|---|
|||
|C-MCA|C-MCA|
|𝑵𝑮|Col2|
|---|---|
|M|LP|
|||
|L|N|
|||
|W-|W-|
|||
|L|N|
|MLP -SAB CAB|Col2|Col3|
|---|---|---|
|M
G-
S-
Conv 3x3|M
G-
S-
Conv 3x3|M
G-
S-
Conv 3x3|
|M
G-
S-
Conv 3x3|Conv 3x3|Conv 3x3|
||||
𝑓� (a) The architecture of SCT for High Dynamic Range imaging (b) Two successive attention blocks
Figure 4. The architecture of our proposed SCTNet for HDR deghosting. (b) The inner structure of two successive attention blocks G-SAB
and S-CAB. The G-SAB leverages the spatial awareness to deal with the intra-image correlation to model the dynamic motion, while our
S-CAB block attends to the channel direction and enables the inter-image intertwining to enhance the semantic consistency across frames.
using the gamma encoding function, as described in Eq. 2:
_e_
_\_ l _[a][b]_ _{_ _e_ q : _h_ i _} H_{i} = \frac {L_{i}^\gamma }{t_{i}}, \quad \gamma =2.2,_ (2)
_l_
where _ti_ is the exposure time corresponding to the image _i_,
_Hi ∈_ R [3] _[×][H][×][W]_ is the gamma-projected input of the corresponding image _Ii_ . By concatenating the LDR images _Li_
with _Hi_, we obtain _Ii ∈_ R [6] _[×][H][×][W]_ for each EV input.


**Shallow Features Extraction** : Given the input _Ii_ _∈_
R [6] _[×][H][×][W]_, we first extract shallow features _fi_ with convolution. Unlike existing methods that compute alignment matrices between the shallow features, we directly merge them
and feed the fused features into our attention blocks.
**Global Spatial Attention Block** : To model the spatial correlation between different frames, we adopt the windowbased multi-head self-attention following [14, 13, 15].
Technically, the merged features are first fed into the patch
partition for tokenization. Then, the sequence of tokens is
fed into the spatial attention block, which contains layer
norm (LN), window-based multi-head self-attention (WMSA), and a point-wise multi-layer perceptron (MLP). For
the _j_ _[th]_ layer, _j ∈{_ 1 _, ..., L}_, it takes the sequence _zj−_ 1 as
input, and outputs the new sequence _zj_ :
_\_ b _e_ g _in_ { _sp_ l _it}_ &\ h _at_ {z
(3)
_}__ j _=_ W _\t_ ex _tn_ or m a _l_ _{-}MSA(LN(z_{j-1})) + z_{j-1}; \\ &z_j = MLP(LN(\hat {z}_j)) + \hat {z}_j. \end {split}_
**Semantic-Consistent Attention Block** : As shown in Figure 4, we propose a semantic-consistent attention block (SCAB) to interact between different frames. Specifically,
taking the spatially refined feature _zj_, we first split it into
three sub-groups _zii_ _[′]_ [, where each group is linked with shal-]
low features of each EV input _fi_ by a residual addition. We
obtain the linked feature _zi_ _[′]_ [by:]
_z_ ['] _[ _]_ _i}_ _[{]_ _[=]_ _[z'_{ii} + f_i. ]_ (4)
Then, we apply the channel-wise multi-head crossattention (C-MCA) between them. Technically, we compute the query from the reference-linked feature, while the
keys and values are generated from others. For simplicity,
we take _z_ 1 _[′]_ [and] _[ z]_ 2 _[′]_ [as examples. Let] _[ q]_ [2] [be the query from]
_z_ 2 _[′]_ [and][ (] _[k]_ [1][;] _[ v]_ [1][)][ be the key and value generated from] _[ z]_ 1 _[′]_ [, the]
output _z_ 12 _[′]_ [of our C-MCA becomes:]
_[=]_
z' [_] _[ {][ 1][ 2} ]_ [=] [C] _[- M]_ [C] _[A (]_ [q][_ 2] _[, k_1, ]_ [v] _[_]_ [1] ~~_s_~~ _[ )]_ t _m_ a _x(\frac {q_2 \times k_1^T}{\sqrt {d_k}})v_1,_
_of_
(5)
where _dk_ is the scaling factor. Similarly, we can obtain the
cross-calibrated feature _z_ 32 _[′]_ [by applying C-MCA on] _[ z]_ 2 _[′]_ [and]
_z_ 3 _[′]_ [. Finally,] _[ z]_ 12 _[′]_ [,] _[ z]_ 22 _[′]_ [, and] _[ z]_ 32 _[′]_ [are merged and fed into an]
FFN to generate the final output of our S-CAB. The computational cost of our C-MCA is:
_\ O meg_ a _(_ [C] _- MC_ A _) = C^2 \times N / 9,_ (6)
where _C_ is the channel dimension and _N_ is the spatial resolution. By extending the inter key-query correlation to
cross-frame key-query correlation, our S-CAB block suggests a natural way to aggregate multi-EV features. Moreover, our cross-attention is applied on the channel axis,
which can deeply preserve and explore the semantic consistency between different frames.
**Skip Connection** : After the attention blocks, we build a
skip connection from the reference feature _f_ 2. The skip
connection is realized through a convolution. Finally, we
use another convolution to generate the final HDR output.


Our network is trained end-to-end. We first supervise
our network with _L_ 1 loss function. Note that Kalantari _et_
_al_ . [10] has shown that computing the loss in the HDR domain leads to the appearance of discoloration effects. To
avoid this, we compute the _L_ 1 loss in the tone-mapped do
Table 2. Quantitative comparison with state-of-the-art methods on the Kalantari [10] testing samples. _l_ -PSNR and _l_ -SSIM are computed in
the linear domain while _µ_ -PSNR and _µ_ -SSIM are computed after _µ_ -law tone mapping (Eq. 7). PU-PSNR and PU-SSIM are computed by
applying the encoding function proposed in [1].
|Method|µ-PSNR PU-PSNR l-PSNR µ-SSIM PU-SSIM l-SSIM HDR-VDP2|
|---|---|
|Method|-PSNR
PU-PSNR
-PSNR
-SSIM
PU-SSIM
-SSIM
HDR-VDP2|
|Hu_ et al_. [7]
Sen_ et al_. [25]
DHDRNet[10]
DeepHDR[31]
AHDRNet[32]
NHDRRNet[33]
CEN-HDR[28]
HDRGAN[19]
HDR-Transformer[15]
**SCTNet** (Ours)|35.79
34.96
30.76
0.9717
0.9615
0.9503
57.05
40.80
40.47
38.11
0.9808
0.9775
0.9721
59.38
42.67
41.83
41.23
0.9888
0.9832
0.9846
65.05
41.65
41.35
40.88
0.9860
0.9815
0.9858
64.90
43.63
42.93
41.14
0.9900
0.9849
0.9702
64.61
42.41
42.97
41.43
0.9877
0.9855
0.9857
61.21
43.05
43.24
40.53
0.9908
0.9821
0.9856
64.34
43.92
44.03
41.57
0.9905
0.9851
0.9865
65.45
44.32
44.23
42.18
0.9916
0.9924
0.9884
66.03
**44.49**
**44.27**
**42.29**
**0.9924**
**0.9927**
**0.9887**
**66.65**|
main using the _µ_ -law function:
\ _l_ a b _[el ]_ [{e q] _[ :m]_ [u] = \fr _ac {log(1+\mu H)}{log(1+\mu )}, \quad \mu =5000,_ (7)
_law_ } T _(_ H _[)]_
where H is the linear HDR image and _µ_ is the amount of
compression. Our _l_ 1 loss for HDR imaging becomes:
\ _la_ b _e_ l _{_ e _:_ [q] l _1_ } _\mathcal {L}_1 = \lVert T(H) - T(\hat {H}) \rVert _1,_ (8)
where _T_ is the _µ_ -law tone mapping function, _H_ and _H_ [ˆ] are
the prediction and the ground truth in the HDR domain.
Additionally, following [15], we use an auxiliary perceptual loss _Lp_ for supervision. The perceptual loss measures the difference between the feature representations of
the output image and the ground truth image at multiple layers of the pre-trained CNN, realized by computing the mean
squared error between the feature maps at each layer. The
motivation for such a loss is to assess the dissimilarity between the predicted HDR image and the actual ground truth
image. Formally, we have our perceptual loss:
\ m _t_ hc _a_ l {L _}_ _p = _\_ su _m_ __{j} \lVert \Phi (T(H)) - \Phi (T(\hat {H})) \rVert _1,_ (9)
_a_
where Φ are the feature maps extracted from a pre-trained
VGG-16 network with _j_ indicating the _j_ -th layer. Finally,
the training loss can be formulated as:
_\_ la _be l {_ e q _: l os_ s _fu nct_ i _on} \mathcal {L}= \mathcal {L}_1(T, T_{GT}) + \alpha \cdot \mathcal {L}_p(T, T_{GT}),_ (10)
where _α_ is a hyper-parameter set to 0.01.


**Datasets** : We first evaluate our model on the conventional
Kalantari [10] benchmark composed of 74 training scenes
and 15 testing scenes. We follow the common practice of
cropping patches of size 128 _×_ 128 with a stride of 64 from
the training set. We use rotation and flipping as data augmentation. We also evaluate our proposed dataset which is
composed of 108 training samples and 36 testing samples.
Each sample represents the acquisition of a dynamic scene
caused by large foreground or camera motions. The same
dataset preparation and augmentation methods are applied.
**Implementation overview** : We implement our SCTNet on
PyTorch with ADAM optimizer with an initial learning rate
of 2e-4, _β_ 1 to 0.9, _β_ 2 to 0.999, and _ϵ_ to 1e-8. We train the
network on 2 V100 GPUs. All the counterparts’ results are
trained or downloaded from the official resources.
**Metrics** : We use common metrics such as PSNR and SSIM,
as well as HDR-VDP2 metric which was specifically designed for the HDR task. We compute PSNR and SSIM
evaluation in linear and tone-mapped domains, denoted as
_l−_ and _µ−_, respectively. We also compute PU-PSNR/SSIM
in the tone-mapped domain as proposed in [1]. We set the
display peak luminance to 100cd/m², a 1000:1 contrast, and
an ambient light to 10 lux.


**Evaluation on Kalantari** _**et al**_ **. Dataset** : We present the
quantitative comparison in Table 2. We compare our
method with 9 widely-used HDR methods, varying from
early hand-crafted, lightweight, CNN, and Transformer
methods. Specifically, Hu _et al_ . [7] and Sen _et al_ . [25]
are based on input patch registration methods. DHDRNet [10] is one of the pioneering learning-based methods
that leverage the optical flow to align the inputs. Similarly,
DeepHDR [31] aligns the background feature by the homography and then feeds the aligned inputs into the classical encoder-decoder architecture. NHDRRNet [33] further
leverages the non-local attention module during the learning stage. AHDRNet [32] is the first method that introduces an attention block for feature alignment at the shallow stage. Then, the calibrated features are merged and fed
into multiple dilated residual dense blocks for feature encoding. HDRGAN [19] is the first GAN-based approach
LDR inputs Our tonemapped HDR image LDR patches Kalantari et al. DeepHDR AHDRNet HDR-GAN HDR-transformer Ours Ground truth
Figure 5. Comparison of our proposed network with state-of-the-art methods on a Kalantari dataset’s sample [10]. Thanks to the red
patches, we show that our method is the most efficient to recover the ghost from the foreground motion in a bright area of the scene. With
blue patches, we exhibit the ability of our solution to recover faithful information in occluded regions. Additionally, as demonstrated in
both red and blue patches, our method excels in retrieving the building’s texture information. Please zoom in for more details.
Table 3. Quantitative comparison on our dataset. All the models
are trained through official implementations. Our method signifi
|tly outperforms all counterparts by a large margin.|Col2|
|---|---|
|Method
_µ_-
PSNR
_l_-
PSNR
_µ_-
SSIM
_l_-
SSIM
HDR-
VDP2|Method
_µ_-
PSNR
_l_-
PSNR
_µ_-
SSIM
_l_-
SSIM
HDR-
VDP2|
|NHDRRNet [33]
DHDRNet [10]
AHDRNet [32]
HDR-Transf. [15]
**SCTNet** (Ours)|36.68
39.61 0.9590 0.9853 65.41
40.05
43.37 0.9794 0.9924 67.09
42.08
45.30 0.9837 0.9943 68.80
 42.39
46.35 0.9844 0.9948 69.23
**42.55**
**47.51 0.9850 0.9952 70.66**|
Inputs HDR Image Input patches
DHDRNET AHDRNET HDR-Transformer Ours GT
Figure 6. Qualitative comparison on our proposed dataset. All the
methods are trained using our training set. Our network outperforms all the SOTA counterparts in dealing with ghost artifacts
due to the foreground motion. Please zoom in for details.
for HDR merging with a deep supervised HDR method,
while HDR-Transformer [15] is the first work by replacing
the conventional convolution operations by the transformer
architecture. It can be seen that, with the standardized evaluation protocol, our method performs favorably against all
the counterparts, setting new records on the conventional
Kalantari _et al_ . [10] dataset. A qualitative comparison can
be found in Figure 5. It can be seen that our method can
lead to prediction closer to the ground truth.
**Evaluation on Our Dataset** : We also conduct comparisons with the most representative HDR methods on our
proposed dataset. In Table 3, it can be seen that our method
Table 4. Computational cost comparison of the proposed SCTNet
solution against SOTA methods. The number of operations and
parameters are evaluated using the script provided by the NTIRE
[22] challenge. The input size is set to 1000 _×_ 1500 pixels. The
speed is measured on a Quadro P6000 GPU.
|Method|Params. (M)GMACsSize (Mb)Speed (s)|
|---|---|
|Method|Params. (M)GMACsSize (Mb)Speed (s)|
|DeepHDR [31]
AHDRNet [32]
HDR-GAN [19]
HDR-Transf. [15]
**SCTNet** (Ours)|16.61
1453.70
66.5
0.32
1.52
2166.69
**6.1**
0.30
2.63
778.81
10.6
**0.29**
1.22
981.81
53.4
8.65
**0.99**
**293.77**
29.1
7.14|
sets new records on almost all the metrics, approving our
versatility and effectiveness across different datasets. We
provide qualitative results of our method in Figure 6. It can
be seen that our method can better deal with object movement and generate the HDR output closer to the ground
truth. This can be attributed to our attention blocks focusing
on inter-image consistency.
**Generalization Capability** : We provide in Figure 7 the visual comparison on two unseen scenarios. We can observe
that our method achieves consistently superior performance
compared to all SOTA counterparts. Firstly we lead to better
performance in restoring the illumination gradient resulting from the fire motion as shown in Figure7-Left. While
dealing with foreground motion as shown in Figure7-Right,
Our method outperforms others in recovering the maximum
amount of information from the bright areas of the scene.
These observations further validate our effectiveness.
**Comparison on the Computational Cost** : The computational cost of our method, as well as those of our counterparts, can be found in Table 4. Our method achieves the best
trade-off between the performance and the computational
cost. It is worth noting that compared to the current SOTA
HDR-Transformer [15], our method only requires around
1/3 of the GMACs and 1/2 of the model size, validating our
efficiency on attention blocks. Compared to AHDRNet, as
shown in Table 2 & 4 we achieve a notable improvement of
+1dB on PSNR while using only 13% of the GMACs and
65% of the parameters, showcasing our efficiency.
Inputs Our tonemapped prediction Input patches
AHDRNET HDR-GAN HDR-Transformer Ours
Inputs Our tonemapped prediction Input patches
AHDRNET HDR-GAN HDR-Transformer Ours
Figure 7. We analyze the generalizability of our solution by evaluating on the unsupervised dataset proposed by Sen _et al_ . [30]. For a
fair comparison, all networks are trained using our proposed dataset. In the first comparison (left), our network shows superior ability
in restoring the illumination gradient resulting from the fire motion. In the second comparison (right), our method outperforms all the
counterparts in recovering the maximum amount of information from the bright areas of the scene. Please zoom in for more details.
Table 5. Ablation study on the key components. The comparison
|Base. G-SAB S-CAB Skip|µ-PSNRPSNR µ-SSIM SSIM|
|---|---|
|✓
✓
✓
✓
✓
✓
✓
✓
✓
✓|43.42
41.68
0.9882
0.9861
44.04
41.89
0.9921
0.9887
44.12
41.71
0.9923
0.9882
**44.49**
**42.29**
**0.9924**
**0.9887**|


**Key Component Analysis:** In this section, we conduct
studies to verify the effectiveness and importance of our
proposed components: G-SAB, S-CAB, and Skip connection to link with the shallow multi-EV features. The experimental results can be found in Table 5. We gradually add
our blocks on top of our baseline (denoted as Base.). It can
be seen that each component brings improvement, validating the effectiveness of our proposed method. More ablation studies can be found in the supplementary material.
**Attention Visualization:** To better understand our proposed attention blocks, we provide feature visualizations
between _I_ 1 and the reference frame _Iref_ in Figure 8. It can
be seen that: **(1)** Our G-SAB leverages global clues, such
as shape and contour, while the S-CAB focuses on feature
alignment across input frames. **(2)** The S-CAB leverages
semantic clues to compute cross-image affinities, leading
to activated attention on misaligned areas, as shown by the
**green** bounding boxes. **(3)** The complementary and simultaneous operation of our blocks leads to improved feature
modeling for effective deghosting.


In this paper, we present a novel alignment-free network
with a Semantics Consistent Transformer for HDR deghost
Figure 8. _I_ 1- _Iref_ attention visualization.
|Col1|Col2|
|---|---|
|_S-CAB_
|_S-CAB_
|
ing. Different from existing methods that estimate the image alignment by attention matrix at the shallow stage, our
method is unified in that we jointly model the foreground
motion and static semantic context simultaneously. We propose two types of attention blocks with intra-/inter-image
interaction to directly deal with the alignment issues in the
transformer architecture. Furthermore, we propose a novel
dataset with more realistic samples and more variations on
both lighting conditions and scene motions.
**Future work** . There are several directions. First, although
our method sets new records on both the existing Kalantari benchmark and our dataset, there is room for further
improvement, especially in further reducing the computational cost while maintaining the performance. A second
direction can be on the domain adaptation. How to improve
the performance on one dataset while being trained with the
other is yet an open question. We hope our dataset can contribute to the development of methods of better adaptation
and generalization capability.
**Ackowledgement** The authors thank the anonymous reviewers and ACs for their tremendous efforts and helpful
comments. This research is financed in part by the Conseil
R´egional de Bourgogne-Franche-Comt´e and the Alexander
von Humboldt Foundation.


_Simon Durand_ _[1*]_ _, Daniel Stoller_ _[1*]_ _, Sebastian Ewert_ _[1]_
1Spotify *Equal contribution

Contrastive Learning-Based Audio to Lyrics Alignment for Multiple
  Languages

Simon Durand
Daniel Stoller
Sebastian Ewert


Lyrics alignment gained considerable attention in recent years. Stateof-the-art systems either re-use established speech recognition toolkits, or design end-to-end solutions involving a Connectionist Temporal
Classification (CTC) loss. However, both approaches suffer from
specific weaknesses: toolkits are known for their complexity, and
CTC systems use a loss designed for transcription which can limit
alignment accuracy. In this paper, we use instead a contrastive learning procedure that derives cross-modal embeddings linking the audio
and text domains. This way, we obtain a novel system that is simple
to train end-to-end, can make use of weakly annotated training data,
jointly learns a powerful text model, and is tailored to alignment. The
system is not only the first to yield an average absolute error below
0.2 seconds on the standard Jamendo dataset but it is also robust to
other languages, even when trained on English data only. Finally,
we release word-level alignments for the JamendoLyrics Multi-Lang
dataset.
_**Index Terms**_ **—** lyric alignment, music information retrieval,
audio signal processing, speech recognition, open-source dataset


For many years, the accuracy of systems for the automatic alignment
of audio and lyrics content was well below the requirements for
practical applications [1–5]. However, since 2019 there has been a
resurgence of research and results improved by an order of magnitude.
In this context, given an audio recording and textual lyrics for a piece
of music, the goal of automatic lyrics alignment is to identify for
each lyrical element the corresponding position in the audio. Hereby,
lyrical element can refer to paragraphs, lines, words, phonemes or
even characters, depending on the need for temporal precision.
Current systems typically either employ toolkits designed for
speech recognition or neural network-based end-to-end systems, both
with specific advantages and disadvantages. The former represent a
working recipe for many complex systems often centered around a
hidden Markov model (HMM) or transducer that transfers a lot of
past experience from speech recognition over to lyrics processing.
This can yield state of the art results [6] and tends to maximize the
usefulness of small datasets. However, using tightly integrated components as well as task- and language-specific adaptations based on
expert knowledge [7–10] such as phoneme dictionaries and duration
models, such systems are often considered cumbersome to train and
to be limiting experimentation and further progress. Since the release
of larger scale noisy datasets [11,12] and methods to clean them [13],
many efforts [14,15] have focused on building conceptually simpler
end-to-end systems that enable quick iteration and can make use of
larger amounts of data. Many of these systems are trained using the
Connectionist Temporal Classification (CTC) loss [16], which is a
specific instance of the forward-backward (FB) algorithm [17].
CTC-based training was first employed in [18] for lyrics alignment and was further explored in [19–21]. While CTC enables training neural networks from unaligned lyrics-audio pairs and thus from
larger datasets, there are several drawbacks to using CTC for alignment purposes. First, CTC acts as a transducer by introducing an _ϵ_
symbol (see [16] for details), which means ’no output’ for an audio
frame. It was originally used to enable HMM-less decoding of neural
network outputs for transcription, but can lead to ill-defined target
paths when used for alignment. Second, CTC assumes the text and
audio sequences match one-to-one except for temporal distortion and
thus it is not prepared for missing or extra symbols in the annotation a problem more severe for lyrics than speech due to the prominence
of unannotated instrumental sections and general annotation errors.
Third, CTC at training time is linear in the number of symbols it
supports, which means that in practice CTC systems only support
characters or phonemes but not (sub-)words - yet these could be
detected more robustly. As a consequence, CTC does not support
context-dependent phones while the surrounding context within a
word affects phonetic variability greatly.
Inspired by embedding-based approaches in neighbouring
fields [22–25], we present a novel end-to-end method that represents
the audio signal and the lyrics as sequences of embeddings and uses
contrastive learning to map them into a joint space without requiring
additional information. This way, we obtain a stable loss function
that does not require an _ϵ_ symbol or a fixed in-memory state space,
allowing us to model characters in context with their neighbours,
which would otherwise lead to an infeasibly large vocabulary due to
combinatorial explosion. Due to this data-driven, neural text representation, our system is easy to extend to additional languages, and more
robust than modelling each character independently. We only require
an unordered bag of words occurring in an audio segment to train the
system, opposed to conventional methods that require an ordered,
complete word sequence or even symbol timings. It performs at the
level of the state-of-the-art, yielding an average absolute error below
0.2 seconds on the standard JamendoLyrics dataset [18] for the first
time. We also release manually annotated word-level alignments
for an extension of that dataset featuring three additional languages.
This enables a robust evaluation of multi-language lyrics alignment
approaches, whereas prior work in this setting [19, 26] relied on
performance estimates from noisy annotations [11], or required
duplicate songs.
In the following section 2, we will present the core elements of
our approach: the text and audio encoders, the similarity matching
and training procedure, and the line-based decoding. Section 3 shows,
via an evaluation on the extended JamendoLyrics Multi-Lang dataset,
the relative advantages of our main design choices, and how the
approach is both competitive with the state-of-the-art and can scale
well to additional Latin languages. Finally, we conclude and present
future directions in section 4.
Similarities
Token embeddings
|0.8 0.4 0.2 0.5
0.3 0.9 0.0 0.1
0.1 0.2 0.4 0.2
0.1 0.1 0.3 0.1
0.1 0.1 0.1 0.1|Col2|Col3|Col4|
|---|---|---|---|
|0.3
0.1
0.1
0.1
0.8|0.4|0.2|0.5|
|0.3
0.1
0.1
0.1
0.8|0.9|0.0|0.1|
|0.3
0.1
0.1
0.1
0.8|0.2|0.4|0.2|
|0.3
0.1
0.1
0.1
0.8|0.1|0.3|0.1|
|0.3
0.1
0.1
0.1
0.8|0.1|0.1|0.1|
|||||
||V
A|V
A||
||V
A|A|C|
**Fig. 1** . Our proposed approach training for an audio-lyrics pair. Positive
lyrics tokens are sampled from the lyrics, while negative ones are taken from
lyrics of other songs in the dataset. The audio and text encoders produce
embeddings for the audio and lyrics tokens, respectively. Finally, for each
lyrics token embedding, the maximum similarity to the audio embeddings is
maximised (positive examples) or minimized (negative examples).


An overview of our proposed similarity model and its training is given
in figure 1. In the following, we describe its audio and text encoder
components, the contrastive learning procedure for training, and the
alignment decoding.


The first component of our similarity model is the audio encoder _fa_,
which aims to detect the phonetic content of the singing voice in
the audio. It receives features **X** _∈_ **R** _[T][ ×][D]_ that represent the audio
content with _D_ -dim. feature vectors over time _t ∈{_ 1 _, T_ _}_ . In this
paper, we use spectrogram representations with 5 seconds duration,
magnitudes normalized by _y →_ log(1 + _y_ ), a sampling rate of 11025
Hz, an FFT size of 512 and a hop size of 256. Other representations
are also possible, as long as their temporal resolution is high enough
for subsequent alignment. For the encoder itself, we use a residual
network with 10 residual convolutional blocks (RCBs). Each RCB
consists of 2 repetitions of the following sequence of operations:
group normalization [27], ReLU activation, and a 2D convolutional
layer with a (3x3) kernel and 64 features. The output of the above
is added to the input to the RCB. Finally, a 1D convolution layer is
applied on each time bin with _E_ filters of kernel size _D_ to eliminate
the frequency dimension and yield an embedding matrix **A** _∈_ **R** _[T][ ×][E]_
with _E_ = 64 as the embedding size. This output, with a purposely
small receptive field of 930 ms, is supposed to describe the local
phonetic content of the singing voice in the audio to enable matching
with the lyrics text.


With an estimate of how the singing voice could sound for any part
of the given lyrics text, we can match the text parts to the representation from the audio encoder. This estimation is performed by our
text encoder _fℓ_, which receives the lyrics text as a sequence of symbols _s_ 1 _, . . ., sN_, where the symbols could correspond to characters,
phonemes or other text representations. To process these symbols,
we pass them through a trainable embedding layer that maps each
symbol to a different embedding vector, resulting in an embedding
matrix **L** _∈_ **R** _[N]_ _[×][E]_ .
The resulting embeddings describe the likely phonetic content independently for each symbol. However, the pronunciation
of a symbol usually depends on the neighbouring symbols in
the text, especially when using characters as symbols. Therefore, we extend the text encoder to process the subsequence
( _sn−C_ _, sn−C_ +1 _, . . ., sn, . . ., sn_ + _C_ ) for each symbol _sn, n_ _∈_
_{_ 1 _, . . ., N_ _}_, containing the _C_ previous and following symbols
as context information. A special padding symbol is used for _sn_ with
_n <_ 1 or _n > N_ to ensure an equal length of each subsequence.
In case we know the language of the input song [1], the text encoder
could exploit this information to better estimate the pronunciation. In
this language-conditioned setting, we create a trainable embedding for
each language and append it to each text subsequence. In this paper,
we use a simple network with one fully connected layer (three in case
of language conditioning) with ReLU activation, and a linear output
layer to yield an _E_ -dim. embedding for each symbol – although more
complex architectures can be used for further improvements. Both
the text and audio encoder embeddings are L2 normalized to enable
cosine similarity comparisons.


The audio and text encoders project the audio and lyrics into a shared
embedding space. We aim to train the model such that these similarities are high for matching audio and text parts and low for nonmatching ones. If we knew the exact position of each symbol in the
audio, this could simply be achieved by training a classifier where
the correct symbol for each position in the audio has to be identified,
and each class is equal to one (unique) symbol from the current lyrics
sequence. However, such strong labels are difficult to obtain with
sufficient accuracy, so we consider the case where we only know the
start and end of the lyrical lines in the audio.
In this weak label case, we cannot apply such a simple classification objective. We also do not want to use an auto-regressive
sequence-to-sequence model objective [29], as it incentivizes a strong
language model, which we found in our early experiments to be
detrimental to alignment if it inhibits the acoustic model. Instead,
we apply contrastive learning, where a positive example _s_ [+] is taken
from the lyrics for a given audio segment and negative examples _s_ _[−]_
are sampled from the distribution _ps_ over symbols obtained from
all lyrics in the dataset that do _not_ [2] appear in the audio segment.
To estimate whether a symbol _s_ appears somewhere in the audio,
we take its maximum similarity over the whole audio features **X**,
_m_ ( **X** _, s_ ) = max _t fℓ_ ( _s_ ) _· fa_ ( **X** ) _[T]_ _t_ [, and apply the training objective:]
_L_ = E( **X** _,s_ +) _∼pd_ �( _m_ ( **X** _, s_ [+] ) _−_ 1) [2] + E _s−∼ps_ _m_ ( **X** _, s_ _[−]_ ) [2][�] (1)
where _pd_ is the distribution over audio examples and symbols sampled
from the corresponding lyrics sequence. The loss drives the maximum
similarity between positive audio-symbol pairs up to 1. Negative
pairs are pushed towards 0 to make such audio and text embeddings
orthogonal.
1As lyrics are part of the input, this is a reasonable assumption. For
instance, there is 100% accuracy on JamendoLyrics Multi-Lang using [28].
2If unannotated symbols are present in the audio and mistakenly used as
negatives, our method still works as the majority of examples are correct.


After training the model on pairs of audio and text fragments, we
perform the alignment by computing a normalised similarity matrix between audio and text sequences: **S** = [1] [(] **[A]** _[ ·]_ **[ L]** _[T]_ [ + 1)][, with]
trix between audio and text sequences: **S** = [1] 2 [(] **[A]** _[ ·]_ **[ L]** _[T]_ [ + 1)][, with]
**S** _∈_ [0 _,_ 1] _[T][ ×][N]_ . We then decode an alignment from **S** by finding a
monotonic path maximizing the cumulative similarity score.
With the above decoding, we found our model to mistakenly
position the last few words of a lyrical line close to the start of the
next one (and similarly, the first few words of a lyrical line at the
end of the previous one). To alleviate this problem, we exploit that
lyrics are temporally clustered and constrain the model to output
all words from the same line close to each other. From the initial
alignment obtained with **S**, we estimate each line interval as starting
at _ts_ = _tc −_ _[t][d]_ 2 _[−][d]_ and ending at _te_ = _tc_ + _[t][d]_ 2 [+] _[d]_ . Here, _tc_ represents
the center time of the line by taking the estimated start time of its
middle token, which we found to be robust to outliers. _td_ is the
duration of the lyrical line, estimated by multiplying the number of
tokens in the line with an estimated duration _d_ per token (which we
empirically set to 0 _._ 2s for characters and 0 _._ 4s for phonemes but could
be made token- and song-dependent to improve performance).
Using that estimated line position, we define a line-mask **M** _∈_
[0 _,_ 1] _[T][ ×][N]_ for **S**, with each column of **M** constructed as shown in
figure 2 and constraining each token to be aligned around its estimated
line position. Note that **M** does not require any additional training
or external system, as opposed to using an external vocal activity [9]
or boundary [21] detection module, and is really fast to obtain. The
final alignment is produced by applying the initial decoding scheme
again, but this time to the masked similarity matrix **S** _◦_ **M**, with _◦_
as the Hadamard product. As a result, high values within the line, as
seen in figure 2, encourage the model to position all the tokens of a
line within the initially estimated line interval ( _ts, te_ ). To account for
errors in the initial estimation, a linear tolerance window around the
line borders is used. We empirically set the length of this window to
2 _._ 5s, but find performance to be quite robust to this parameter.
2 _[−][d]_ and ending at _te_ = _tc_ + _[t][d]_ 2 [+] _[d]_


We use two standard evaluation metrics, the average absolute error
(AAE) in seconds and the percentage of correct onsets according
to a tolerance window of 0.3 seconds (PCO), due to their complementary nature: the AAE, where smaller is better, is sensitive to
both very small and especially very large positioning errors, whereas
PCO, where higher is better, encourages predicting timestamps that
are acceptably close to the ground truth while treating all types of
errors the same. For training, we use a dataset of professional-quality
recordings featuring English, Spanish, German and French songs.
Our models are trained for at most 100 epochs with 20,000 iterations
each, using an ADAM optimizer with learning rate 0 _._ 001. An epoch
contains approximately 400 hours of data, sampled among a collection of 87,785 songs. We reserve 2% of our data for validation and
compute the loss after each epoch for early stopping with a patience
of 20 epochs. Our model has 1.2 Million parameters for a modest
total size of 4.8MB. All our phoneme experiments are based on the
CMU English pronunciation dictionary [3] . For each example, and
we use all symbols as positive samples together with 1000 negative
samples.
We use the openly available JamendoLyrics Multi-Lang dataset [30] for evaluation, that features Creative Commons songs and is
an extension of the JamendoLyrics dataset [18] to German, French,
[3http://www.speech.cs.cmu.edu/cgi-bin/cmudict](http://www.speech.cs.cmu.edu/cgi-bin/cmudict)
**Fig. 2** . Constructing one row of the line-mask **M** for the lyrical line “Here you
are” (red line), used during line-based decoding. Dashed arrows correspond to
the character’s start time estimated in the first decoding. The initial estimates
for “h” in “here” and “re” in “are” are moved close to the other characters in
the line by the line-based decoding.
and Spanish (60 additional songs with a high genre diversity) on top
of the existing English data. We extend this dataset with manually annotated word-level timings for all songs. We do not evaluate on other
commonly used datasets as they are smaller, only support English
recordings and might overlap with our training set.


We start by evaluating different configurations of our system, to
assess the impact of the loss, contextual embeddings, the choice of
text representation, and the use of line-based decoding. We then
compare our approach with the current state-of-the-art and conclude
by investigating the multilingual performance.
_How does CTC compare to our contrastive learning approach?_
To evaluate against a CTC approach, we build a CTC-based model
using the audio encoder from our similarity model, leaving the data
and experimental setup unchanged. We add a fully connected layer
to the audio encoder output to get phonemes probabilities, and use
the CTC loss during training. As shown in Table 1 (model M1),
the performance is similar to the CTC approach from [18] shown in
Table 2. However, its AAE is significantly worse than our baseline
similarity model (M3) using the same, phoneme-based configuration.
Our proposed text encoder together with contrastive learning then
appears to yield better performance than a simple CTC approach.
In the above comparison (between models M1 and M3), no additional prior knowledge is used to constrain the models for further
performance improvements. However, high-performance CTC systems [19, 21] are indeed incorporating additional information by
adding vocal separation, pitch information or external boundary estimation. Similarly, we also obtain a competitive AAE for the CTC
approach (M2) when swapping the mask **M** used for line-based decoding with the one obtained from our best similarity model. This
suggests CTC approaches rely heavily on such additional constraints,
whereas our model is more robust while being comparatively simple.
_What is the impact of a contextual text representation?_ We can
compare similarity models processing multiple symbols with the text
encoder (models M5 and M6) and those processing a single symbol without additional context (models M3 and M4) in Table 1. We
see that processing multiple (M5) rather than single (M3) phonemes
leads to significantly stronger results [4] . This effect is even stronger
for characters, where processing characters independently (M4) leads
to poor performance, and using multiple characters (M6) yields an
AAE of 0.15s, the lowest reported in the literature so far. The above
results show the benefits of modeling a symbol based on contextual information influencing its pronunciation. We hypothesize this
contextual information also leads to less spurious activations by nonvocal sounds. For instance, individual symbols are more easily mistaken with an instrumental section than sequences of symbols. It may
4We use small values for _C_ as long-range text dependencies are not crucial
for determining the sound of each symbol.
Model Loss Token _C_ LB-Dec. AAE PCO
M1 CTC Phon 0 ✓ 0.90 86
M2 CTC Phon 0 ✓ [*] 0.20 89
M3 Sim Phon 0 ✓ 0.39 89
M4 Sim Char 0 ✓ 1.60 73
M5 Sim Phon 1 ✓ 0.16 **93**
M6 Sim Char 1 ✓ **0.15** 92
M7 Sim Char 1 ✗ 0.24 91
  - Uses mask matrix **M** from the best similarity model
**Table 1** . Different model configurations trained on English data only and
evaluated on the JamendoLyrics English dataset, using the CTC or similarity
(“Sim”) model, a character (“Char”) or phoneme (“Phon”) representation, and
_C_ context symbols as described in section 2.2. “LB-Dec.” refers to line-based
decoding from section 2.4.
System Dependencies [*] AAE (s) PCO (%)
SDE [18] None 0.82 85
VHMRA [19] SS 0.37 92
DAD [9] Tr, Pho 0.31 93
HBE [21] SS, Pitch, Pho, Boun 0.23 **94**
GYL [6] Gen, Pro, Mult, LM, Pho 0.22 **94**
M5 Pho 0.16 93
M6 None **0.15** 92
- SS: Source separation. Pho: Grapheme-to-phoneme model. Pitch, Gen:
Uses pitch or genre information during training. Tr: Transcription model. B:
Boundary model. M: Multiple alignment models. Pro: Pronunciation model.
LM: Language model.
**Table 2** . Comparison with published methods on the JamendoLyrics English
dataset, and their dependencies. We see that previously, more dependencies
increased performance, at the cost of additional complexity.
also make negative sampling more discriminative. These contextual
text embeddings cannot be used with CTC approaches because of
a combinatorial explosion when considering all possible sequences
of symbols as individual classes, and because of the limited support
for text modeling. This might be one reason why recent CTC-based
works use external information to yield a reduced search space and
work well. Unlike standard toolkits that can use triphone states, we do
not have to deal with a large increase in vocabulary size. Furthermore,
our approach learns from the data directly so we do not have to manually model context-dependant text pronounciation with heuristics that
are hard to scale to different languages and music styles.
_Is a phoneme representation mandatory?_ If the text encoder does
not process symbols with context, we see that a phoneme representation (model M3) is important to get a good performance, compared
to using a character representation (model M4), as indicated by prior
research [6, 9, 19, 21]. However, when the text encoder processes
characters with context, then a phoneme representation (model M5)
performs similarly as a character representation (model M6). We hypothesize that contextual character representations could be thought
as a form of data-driven and end-to-end phoneme representation.
_What is the effect of a line-based decoding?_ When we skip the
line-based decoding (M7) of the best character model, we observe
a significantly worse AAE. It can indicate that the mask **M** indeed
helps removing outliers. We note that all configurations worked better
using this simple decoding process.
_How do similarity models compare to the state-of-the-art?_ We
compare our results with several published models in Table 2. We see
both the phoneme and character similarity models surpass the stateof-the-art for the AAE metric, and stay close to the state-of-the-art for
All EN ES DE FR
Model Data AAE PCO AAE AAE AAE AAE
M5 EN 1.11 59 0.16 1.72 1.22 1.36
GYL [6] EN 0.65 73 0.22 0.97 0.62 0.78
M6 EN 0.35 89 **0.15** 0.37 0.41 0.47
M6 All 0.29 91 0.39 0.22 0.26 0.28
M6 Lang-Cond. All **0.18** **94** 0.21 **0.13** **0.16** **0.19**
**Table 3** . Evaluation of different systems on the JamendoLyrics Multi-Lang
dataset, training on English data only (“EN”) or on English, Spanish, German,
and French (“All”).
the PCO metric, while requiring at the same time less dependencies.
_How do similarity models perform on multilingual data?_ To
estimate the multilingual performance of published models trained
on English data, we apply the NUS AutoLyricsAlign software [5] on
the JamendoLyrics Multi-Lang dataset, and evaluate the alignment [6] .
We find that the performance on English matches what was published
by the authors [6]. In Table 3, we compare this phoneme model
against our phoneme (M5) and character (M6) variants, also trained
on English data only. We see the phoneme models don’t retain their
strong English performance on other languages while the characterbased variant manages to maintain relatively high performance on
non-English data, with an AAE almost halved compared to [6]. This
highlights the risk of error propagation of phoneme models if we try
to extend the scope to additional languages, and that the performance
is limited by the performance of the external phoneme representation.
As expected, including non-English training data (for model M6)
increases performance on the non-English subsets, but its performance on English decreases. Importantly, adding a language conditioning layer to M6 (last row of Table 3), as described in section 2.2,
enables a strong performance on all languages while remaining competitive with the current state-of-the-art on English data – indicating we successfully leverage knowledge of lyrics language without
language-specific modelling effort. The model sets a new benchmark
for multilingual lyrics alignment.


In this paper, we presented a novel lightweight system tailored to
lyrics alignment featuring an audio and text encoder, and trained
using contrastive learning on weakly labelled data. The alignment
is performed on a similarity matrix of text and audio embeddings,
and the proposed approach reaches the state-of-the-art on English.
We show that processing lyrics characters with a context window in
an end-to-end fashion is key to reach the accuracy of a handcrafted
phoneme representation, and to generalise to unseen languages. With
the addition of a simple and easily usable language-conditioning,
we obtain strong performance across all languages. We share wordlevel alignments for the JamendoLyrics Multi-Lang dataset that researchers can extend to a more diverse set of languages, and use
to evaluate multi-lingual alignment systems rigorously. We believe
our embedding-based, data-driven approach is key to address lowresource languages by means of transfer and few-shot learning, as
the same model architecture can be reused across languages. As our
contrastive learning approach can be trained on audio where the lyrics
overlap in time (e.g. duets, Ad-Lib) since it does not need information
about word order, it would be interesting to explore this direction and
adapt the decoding stage to enable new types of alignment.
[5https://github.com/chitralekha18/AutoLyrixAlign](https://github.com/chitralekha18/AutoLyrixAlign)
6While there exists a multi-language model [19], we could not find a way
to apply it to this dataset, and we want to avoid evaluation on a dataset with
overlapping train and test data and potentially inaccurate annotations.


Steve Phelps [1] and Rebecca Ranson [2]
1University College London, Computer Science, `steve.phelps@ucl.ac.uk`
2University of Essex, Psychology, `rjohnse@essex.ac.uk`
July 2023

Of Models and Tin Men: A Behavioural Economics Study of Principal-Agent
  Problems in AI Alignment using Large-Language Models

STUDY OF PRINCIPAL-AGENT PROBLEMS IN AI ALIGNMENT USING LARGE-LANGUAGE MODELS


AI Alignment is often presented as an interaction between a single designer and an artificial agent in
which the designer attempts to ensure the agent’s behavior is consistent with its purpose, and risks
arise solely because of conflicts caused by inadvertent misalignment between the utility function
intended by the designer and the resulting internal utility function of the agent. With the advent of
agents instantiated with large-language models (LLMs), which are typically pre-trained, we argue
this does not capture the essential aspects of AI safety because in the real world there is not a oneto-one correspondence between designer and agent, and the many agents, both artificial and human,
have heterogeneous values. Therefore, there is an economic aspect to AI safety and the principalagent problem is likely to arise. In a principal-agent problem conflict arises because of information
asymmetry together with inherent misalignment between the utility of the agent and its principal,
and this inherent misalignment cannot be overcome by coercing the agent into adopting a desired
utility function through training. We argue the assumptions underlying principal-agent problems are
crucial to capturing the essence of safety problems involving pre-trained AI models in real-world
situations. Taking an empirical approach to AI safety, we investigate how GPT models respond
in principal-agent conflicts. We find that agents based on both GPT-3.5 and GPT-4 override their
principal’s objectives in a simple online shopping task, showing clear evidence of principal-agent
conflict. Surprisingly, the earlier GPT-3.5 model exhibits more nuanced behaviour in response to
changes in information asymmetry, whereas the later GPT-4 model is more rigid in adhering to its
prior alignment. Our results highlight the importance of incorporating principles from economics
into the alignment process.


“ _In the end Macintosh tacitly admitted that the ethical behavior pattern of Samaritan II, which_
_refused to throw itself overboard to save a sandbag, and so took the sandbag to the bottom with it as_
_well, was unsatisfactory. He developed Samaritan III, which not only refused to sacrifice itself for_
_an organism simpler than itself, but kept the raft afloat by pushing the simpler organism overboard._ ”
Michael Frayne, The Tin Men, 1965
Large Language Models (LLMs) such as GPT-3 [1] and T5 [2] are designed to predict the next token in a sequence,
given the preceding context in a massive corpus. This prediction capability allows them to generate human-like text
that can be both creative and informative. However, the output of these models is determined by their training data,
and can potentially include harmful, offensive, or inaccurate content [3]. As an additional training step to mitigate
this issue, ‘Instruct’ models were developed, which are trained to provide responses that are deemed helpful, honest,
and harmless by human labelers [4]. These Instruct models have demonstrated impressive capability in performing a
variety of downstream tasks, such as solving mathematics problems or assisting with programming [5].
Of Models and Tin Men- A behavioural economics study of principal-agent problems in AI alignment using
large-language models
One of the challenges in assessing the capability and safety of LLMs is their opacity; AI safety researchers sometimes
refer to them as _“giant inscrutable arrays of fractional numbers”_ [6]. Since many of their capabilities emerge downstream, we cannot reason about their high-level functioning theoretically from first principles, but rather we have to
resort to empirical enquiry and actual experiments (c.f. [7, 8, 9]). This is analogous to using the scientific method
to understand _human_ cognition. The human brain is similarly opaque, and therefore behavioural psychologists test
their hypotheses using controlled experiments with human subjects. Accordingly, researchers have started to adapt
empirical methods from cognitive psychology [10], and behavioural economics [11, 12, 13, 14, 15, 16, 17, 14], to the
study of large-language models
Although they take the form of pure autoregressive functions, the evolving context window can serve as a kind of
working memory, which can be manipulated both externally and by the model itself to enhance the functionality of the
overall system. For example, prompting the model to provide explanations can improve its reasoning through “chain
of thought” [18], which can be thought of as a kind of meta-cognitive strategy [19]. Moreover, information about the
state of an external task environment can be injected into the context window allowing the system to reason and take
actions in a simulated or real environment. Despite being pre-trained, they can solve some optimisation problems, e.g.
simple calculus problems [20]. Therefore, despite the fact that the pre-trained LLM’s weights are static, it can have
one or more internal _mesa-optimisers_ [21] with objective functions other than next-token prediction. When combined
with ability to inject and track the state of a changing task environment, this allows a large-language model to serve
as a substrate for a higher-level entity that is able not only to solve simple reinforcement-learning problems such as
n-armed bandit problems [22], but can also adapt to opponents in repeated normal-form games such as paper-rockscissors [15]. Therefore, we can use pre-trained LLMs to instantiate goal-oriented _agents_ [23], and deploy them into
the real-world to act autonomously [24]. At the time of writing, this has raised many immediate concerns about their
safety in wider society [25].
In the field of AI safety, the key problem is the alignment between the intended purpose and the actual behavior the
system exhibits [26]. These may not always coincide, as the intended behavior is implicit in the system’s purpose,
yet the actual behavior is subject to the idiosyncrasies of AI training, and deployment into a real-world environment.
Moreover, the environment or the intended behavior may not be apriori fully-understood by the designer. Hence, a
key problem is to ensure the congruence of the actual behavior with the system’s intended purpose despite this apriori
uncertainty.
The discrepancy between the intended and actual behaviors is sometimes referred to as the reward-result gap [27].
This term characterizes the difference between the target reward model, which outlines the intended objectives, and
the reward function that would be learned by a hypothetical perfect inverse reinforcement learning process. Such a
divergence indicates potential challenges in ensuring system behaviors align with human values or intentions.
As discussed, the intended behavior of an AI system cannot be exhaustively specified due to the complexity of realworld tasks which have inherent uncertainty and the inevitable incompleteness of human specification [28]. One
technique for dealing with this is to use machine-learning to generalise from a finite sample of labelled behaviour in
specific situations in order to infer a complete utility function using techniques such as Reward Learning from Human
Feedback (RLHF) [29, 30]. However, this approach is not flawless. The learning process may not always converge to
the ‘correct’ model, and the training data for the reward model can be biased, which can skew the learned objective
function [4].
Misalignment in norms or objective functions between human designers on the one hand, and autonomous artificial
agents on the other, can cause conflict. This is sometimes formulated as a 2-player zero-sum interaction with perfect
information. Drawing on the successes of Alpha-Zero in such games, some researchers argue that artificial agents
will always “win” in a conflict with humans because of their superior cognitive capacity; a common analogy is e.g.
_”10-year-old trying to play chess against Stockfish 15”_ [6].
In reality, however, most interactions are n-player and/or non-zero-sum under conditions of imperfect and asymmetric
information. In order to understand conflict in such settings we need to turn from chess to economics.