This section aims to provide a brief explanation of the evaluation tasks used in Table 1.
_lm-evaluation-harness [25]_ is a framework for evaluating language models on a variety of standardized tasks. We focus on the subset of BoolQ, PIQA, HellaSwag, WinoGrande, ARC-e, ARC-c and
OBQA.
_Vicuna Elo Rank [21]_ leverages LLMs to judge other LLMs and assign them a ranking according to
the ELO system.
_OpenAI Evals [26]_ is a set of general LLM evaluation benchmarks OpenAI released along with the
announcement of GPT-4, and to which OpenAI has asked the community to contribute.
_HumanEval [27]_ is a benchmark OpenAI released along with the announcement of Codex and sets
out to measure "measure functional correctness for synthesizing programs from docstrings".


The political leanings of ChatGPT have been investigated in [39]. We evaluated a model fine-tuned on
OpenAssistant Conversations on a subset of the given tests. Prompts were standardized and multiple
samples were drawn, with majority vote deciding on the final answer for each question. Figure 12
depicts the result. We stress that these are very preliminary results and should not be taken with
9 `[https://github.com/CarperAI/trlx](https://github.com/CarperAI/trlx)`
large degrees of certainty, as the community has yet to find consensus on the exact methodology to
perform such evaluations. We will update this section with improved results in the future. Our limited,
preliminary results show the model trained on OpenAssistant Conversations to be more balanced and
varied in its political leanings than ChatGPT.
Figure 12: Comparison of evaluations on test for political leanings. For original ChatGPT results and
references to tests used, see [39]

OpenAssistant Conversations -- Democratizing Large Language Model
  Alignment

Andreas Köpf
Yannic Kilcher
Dimitri von Rütte
Sotiris Anagnostidis
Zhi-Rui Tam
Keith Stevens
Abdullah Barhoum
Nguyen Minh Duc
Oliver Stanley
Richárd Nagyfi
Shahul ES
Sameer Suri
David Glushkov
Arnav Dantuluri
Andrew Maguire
Christoph Schuhmann
Huu Nguyen
Aleaxnder


Throughout the collection of OpenAssistant Conversations, a large global community has been built,
including an active Discord group, and a GitHub repository with over 200 contributors.
Figure 13 shows the growth of the Discord community throughout the duration of data collection.
Figure 14 shows new commits to the GitHub repository over time.
Figure 15 shows the growth in stars on the GitHub repository over time.
Figure 16 shows popularity of OpenAssistant by YouTube’s videos’ views on the theme over time.
These Figures serve as a strong reminder of what can be achieved by the collective effort of many
volunteers, even in a field where research has thus far been largely monopolized by a small number
of industrial labs.
In addition, by comparing the massive influx of new contributors and subscribers to the emergence of
Open Assistant themed videos, it shows how certain media events have influenced the development.
Figure 13: Discord users in the OpenAssistant group over time.
Figure 14: GitHub commits to the OpenAssistant repository over time.
Figure 15: GitHub stars on the OpenAssistant repository over time.
Figure 16: YouTube’s videos’ views on the OpenAssistant theme over time.


Dataset documentation can be found at `[https://huggingface.co/OpenAssistant/oasst1](https://huggingface.co/OpenAssistant/oasst1)` .


We confirm that we bear all responsibility in case of any violation of rights during the collection of
the data or other work, and will take appropriate action when needed, e.g. to remove data with such
issues.


**Mazda Moayeri** [* 1] **Keivan Rezaei** [* 1] **Maziar Sanjabi** [2] **Soheil Feizi** [1]

Text-To-Concept (and Back) via Cross-Model Alignment

Mazda Moayeri
Keivan Rezaei
Maziar Sanjabi
Soheil Feizi


We observe that the mapping between an image’s
representation in one model to its representation
in another can be learned surprisingly well with
just a linear layer, even across diverse models.
Building on this observation, we propose _text-to-_
_concept_, where features from a fixed pretrained
model are aligned linearly to the CLIP space, so
that text embeddings from CLIP’s text encoder
become directly comparable to the aligned features. With text-to-concept, we convert fixed offthe-shelf vision encoders to surprisingly strong
zero-shot classifiers for free, with accuracy at
times even surpassing that of CLIP, despite being much smaller models and trained on a small
fraction of the data compared to CLIP. We show
other immediate use-cases of text-to-concept, like
building concept bottleneck models with no concept supervision, diagnosing distribution shifts in
terms of human concepts, and retrieving images
satisfying a set of text-based constraints. Lastly,
we demonstrate the feasibility of _concept-to-text_,
where vectors in a model’s feature space are decoded by first aligning to the CLIP before being
fed to a GPT-based generative model. Our work
suggests existing deep models, with presumably
diverse architectures and training, represent input
samples relatively similarly, and a two-way communication across model representation spaces
and to humans (through language) is viable.


The representation spaces of deep vision models are undoubtedly rich in semantic structure. However, these deep
feature spaces are notoriously challenging for humans to interpret, mainly because it is hard for us to digest thousands
of numbers at once. Unlike deep models, which encode
*Equal contribution 1Department of Computer Science, University of University of Maryland [2] Meta AI. Correspondence
to: Mazda Moayeri _<_ mmoayeri@umd.edu _>_, Keivan Rezaei
_<_ krezaei@umd.edu _>_ .
_Figure 1._ Overview. After _aligning_ the representation space of a
given image encoder to a CLIP image encoder, we can compare
aligned representations of images to concept vectors obtained
_directly from text_ (typically, an example set of data is required to
obtain each concept vector; our method is example-free, i.e. O(1)
w.r.t. data collection). Using a GPT-based CLIP decoder, we can
map arbitrary vectors in representation space to text. Our method
yields efficient **interpretability** : we only train _one linear layer_ .
concepts as vectors in high (e.g. _d_ = 2048) dimensional
spaces, humans have developed language to describe the
world around us concisely. In this work, we propose a
method to map text to concept vectors that can be compared
directly to image representations obtained from off-the-shelf
vision encoders trained with _no text supervision_ .
Our method works by _aligning_ the representation space of
a given vision model to the representation space of a CLIP
(Radford et al., 2021) model. By design, the CLIP representation space is shared across jointly trained vision and text
encoders. Thus, CLIP models already have text-to-concept
built in, via the text encoder. To extend this capability to
off-the-shelf models, we propose to learn a mapping between representation spaces. Specifically, we optimize a
function to predict the representation of an image for a target
model (i.e. CLIP) from the same image’s representation for
a source model (i.e. off-the-shelf vision model). We can


_Figure 2._ Qualitative validation of text-to-concept. ImageNet classes are sorted by the average cosine similarity of the CLIP embedding
for “in a tree” to the _linearly aligned_ Dino ResNet representations of images within each class. Highest ranked classes indeed often appear
in a tree, as is evident by the most similar instances. The least similar instances appropriately do not contain the concept.
then map the representations of the off-the-shelf model to
CLIP space, where the aligned features would reside in the
same space as the concept vector for the desired text.
The mapping function, however, may significantly change
the semantics of its input. To prevent this, we restrict the
hypothesis space of our mappings to be _affine transforma-_
_tions_ . Despite their simplicity, we find that linear layers are
surprisingly effective at performing feature space alignment,
even between models with diverse architectures and training
procedures. This observation suggests that despite drastically different approaches to training, diverse models seem
to learn to store information in similar ways. Most notably,
we can align model representations to CLIP, thus extending
CLIP’s text-to-concept abilities to existing models.
Figure 2 visually validates our approach: after encoding
the concept “in a tree” in CLIP space and computing similarity with aligned representations from a self-supervised
ResNet, the classes with the highest average similarity are
reasonable, and images within them with the highest similarity prominently display the concept, while the least similar
class instances do not. Stronger validation of our approach
is found in performing **zero-shot classification using off-**
**the-shelf encoders via text-to-concept.** Models achieve
impressive zero-shot accuracy on many tasks, often being
competitive with a CLIP model that is larger, trained on
many more samples with richer supervision, and most notably, directly optimized to align with the text encoder we
use in text-to-concept. Surprisingly, zero-shot accuracy of
off-the-shelf models surpasses the CLIP in a few cases, particularly for color recognition. While greatly expanding the
use cases for existing models, these zero-shot abilities also
support the belief that deep models learn many more abstract
notions than what they are explicitly trained to know. Textto-concept allows for uncovering and better utilizing the rich
semantics hidden in existing models’ representation spaces.
In addition to zero-shot learning for free, text-to-concept
has several immediate interpretability applications, such
as converting vision encoders to _Concept Bottleneck Mod-_
_els_ (CBMs) (Koh et al., 2020) with _no concept supervision_
_required_ . CBMs decompose inference into a concept prediction step followed by class prediction using a white box
model (i.e. linear head) on concept predictions, so that
the contribution of each concept to the final logit can be
precisely computed. Typically, CBMs require concept supervision in addition to class labels, but with text-to-concept,
we can replace concept predictions with concept similarities,
obtained by comparing aligned representations to the vector
obtained for the desired notion. Then, CBM training reduces
to simply training a linear layer to predict class labels from
pre-computed similarities of aligned image representations
to a set of concept vectors. We illustrate this application on
RIVAL10 data (Moayeri et al., 2022), which has attribute labels, though we only use these labels to verify our zero-shot
concept prediction approach. Indeed, we obtain an AUROC
of 0 _._ 8 for RIVAL10 attribute prediction in the zero-shot manner described, leading to a highly accurate (93 _._ 8%) resultant
CBM with desired interpretability benefits (see Figure 7).
Next, we show text-to-concept can demystify large datasets,
as the distribution of similarities between a bank of
text-to-concept vectors and aligned representations of the
data essentially summarizes what concepts are present,
explaining the data distribution in human terms. This can be
applied to diagnosing distribution shifts, as we can inspect
the shift w.r.t. to human-understandable concept similarities.
For example, when comparing ImageNet to ObjectNet
(Barbu et al., 2019), we can show that the distribution of
similarities for the “indoors” concept shifts dramatically,
capturing the essence of why ObjectNet poses a challenge:
images in ObjectNet were taken in people’s homes. Another
way text-to-concept aids in engaging with large datasets
is via concept-based image retrieval. Using _concept logic_,
we query the image representations for a given model that
satisfy a set of concept similarity thresholds, allowing for


greater human control of the importance of each concept
in the search, yielding reasonable results in finding specific
images in a large corpus.
Lastly, we close the human-machine communication loop
by introducing _concept-to-text_ to directly decode vectors in
a model’s representation space. Our implementation aligns
the model’s space to CLIP, and then leverages an existing
CLIP space decoder (Tewel et al., 2021) that uses a CLIP embedding to guide the output of GPT-2. The existing decoder
was intended for image captioning, though we demonstrate
that its abilities extend to general vectors (i.e. not obtained
from a single image) from non-CLIP models after alignment.
Specifically, we decode the vectors in the classification head
of three ImageNet trained models. We then use a human
study to verify that the decoded captions describe the class
associated with each vector, finding that our simple method
works in over 92% of cases.
Our methods extend the capabilities of multi-modal models
like CLIP to other models that are trained on much smaller
uni-modal datasets and with weaker supervisions. This can
be useful when a model more accurate or smaller than CLIP
for a specific domain is desired, or when training CLIPlike models is infeasible, due to the large corpus of imagecaption pairs needed. Moreover, since our approach can be
applied to interpret any model’s representation space, while
only requiring the training of a linear layer, its potential
impact is very high, as text-to-concept is easy to plug-in and
has a breadth of applications. The implications of our work
are also startling: first, the success of linear representation
space alignment indicates that diverse models ultimately represents inputs relatively similarly. The emergent zero-shot
abilities of existing models suggests an under-utilization of
models we already have. Finally, the synergy we display
between CLIP, GPT, and existing models, coupled with the
ability to communicate across these models and back and
forth with humans, makes the prospect of diverse models
collaborating with minimal tuning very promising.


Our alignment of model representation spaces is related to
stitching, first introduced by Lenc & Vedaldi (2015), who
train linear layers to merge top and bottom chunks of different models, resulting in “franken-CNNs”. Stitching was
revisited by Bansal et al. (2021), who aimed to showcase
how it can be used as a tool to quantify the quality of representations towards learning how to obtain better representations, and Csiszarik et al.´ (2021), who consider different
ways to train stitching layers. We note these works typically
stitch together models of the same architecture, where as
we consider a much more diverse set of models. Also, those
works focus on comparing representations to one another,
while we aim to relate representations to human notions.
Namely, we seek to obtain concept vectors within the representation space of off-the-shelf models from text. Also
known as concept activation vectors (CAVs), Kim et al.
(2018) popularized the study of directions corresponding to
human concepts in deep feature spaces, as well as the sensitivity of model outputs to changes along these directions,
so to interpret deep networks. One limitation is the necessity of example sets of data to define CAVs. More recent
efforts automatically discover CAVs (Ghorbani et al., 2019;
Fel et al., 2022; Zhang et al., 2020), though annotating the
discovered concepts with language is not straightforward,
which motivates our concept-to-text method.
The rise of joint vision-language models like CLIP (Radford
et al., 2021) make it possible to interpret vision space with
text, as well as perform zero-shot classification. Follow
up works leverage CLIP to annotate neural nodes (Oikarinen & Weng, 2022) or to distill failure modes (Jain et al.,
2022) of non-CLIP models. However, they engage probe
datasets or exemplars to communicate with CLIP space,
while our method directly aligns representation spaces. Recently, Moschella et al. (2022) devise a zero-shot method for
communication across representation spaces based on relative positions to anchor points, which supports our claim that
representation spaces are sufficiently equivalent to where
the extension of CLIP’s inherent text-to-concept ability to
off-the-shelf models via linear alignment is viable.


We use _X_ to denote the set of all possible input images.
Let _D_ train _, D_ test denote the training and test datasets.
We define a _vision encoder ⊂X_ as a model _f_ that maps images
_x ∈X_ to vectors _f_ ( _x_ ) _∈_ R _[d]_ . Given two vision encoders
_fs, ft_, _representation space alignment_ of model _fs_ to model
_ft_ is the task of learning a mapping _h_ : _fs_ ( _X_ ) _→_ _ft_ ( _X_ ).
We restrict _h_ to the class of affine transformations, i.e.,
_hW,b_ ( _z_ ) := _W_ _[T]_ _z_ + _b_ .
To maximally retain the original semantics of representation
spaces, we design the following optimization problem
The above optimization can be viewed as multiple linear
regression problems; thus we evaluate the linear alignment
on _D_ test by considering the quality of the solution on those
linear regression problems. We use _Coefficient of Determi-_
_nation_, i.e., _R_ [2] which is the proportion of the variation in the
dependent variables that is predictable from the independent
variables. Furthermore, we note that for the vision encoder
_fs_, there usually exists a _classification head gs_ : R _[d]_ _→C_
that classifies a representation in the space of model _fs_ . Indeed, the predicted label for input _x_ is _gs_ ( _fs_ ( _x_ )). Note
_W, b_ = arg min
_W,b_
_D_ train
_|_ _|_
- _∥W_ _[T]_ _fs_ ( _x_ ) + _b −_ _ft_ ( _x_ ) _∥_ 2 [2] _[.]_
_x∈D_ train


Sup
ResNets
Robust
ResNets
Sup
ViTs
SS
ResNets
SS
ViTs
CLIPs
1.00
0.75
0.50
0.25
|0.98|1.03|0.82|0.86|0.79|0.80|
|---|---|---|---|---|---|
|0.87|0.98|0.76|0.80|0.73|0.73|
|1.01|1.08|0.96|0.93|0.86|0.68|
|0.88|0.94|0.77|0.90|0.78|0.79|
|0.93|0.97|0.83|0.81|0.88|0.82|
|0.72|0.76|0.58|0.55|0.46|0.80|
Sup
ResNets
Robust
ResNets
Sup
ViTs
SS
ResNets
SS
ViTs
CLIPs
Target
_Figure 3._ Heatmap of retained accuracy. the value in row _r_ and
column _c_ is the average of retained accuracy when doing alignment
from all models in group _r_ to all models in group _c_ . Note that
“Sup” refers to supervised while “SS” refers to self-supervised
models. All models except CLIPs are trained on ImageNet.
that _C_ denotes the set of labels, e.g., ImageNet classes. We
define _aligned accuracy_ as the accuracy of classification
on _D_ test when we use _fs_ as the vision encoder, then do the
linear transformation to obtain the corresponding representation in space of _ft_, and finally, use _gt_ for classification. If
alignment works well, aligned accuracy should be admissible and comparable to the accuracy of model _ft_ when no
alignment is used. Then, we define _retained accuracy_ as
the ratio of aligned accuracy to the accuracy of model _ft_
without any alignment. Note that we use ImageNet-1K train
and test datasets as _D_ train and _D_ test in linear alignment.
Interestingly, we observe that simple linear alignment works
well in terms of both _R_ [2], and aligned accuracy across various models. Figure 3 shows the aligned accuracy between
diverse pairs of models. In the scope of linear alignment,
we further consider the sample efficiency of optimizing linear alignment as well as investigating linear alignment in
space of top principal components of representation spaces,
where we roughly see strong, near identity correspondence
between the top principal components of different models.
We refer to Appendix A and B for more details.
According to Figure 3, various models are highly alignable
to CLIP models. This is surprising as CLIP models are
trained on other datasets than ImageNet and their training
procedure involves vision/text supervision which is drastically different from other models. High-quality alignment
to CLIP representation space enables models to adopt a
wide variety of CLIP models capabilities, which we analyze
in this work. On the other hand, we observe that retained
accuracy when aligning CLIP models to other models is not
high. This is mainly due to the fact that CLIP models encode images and texts in relatively low-dimensional spaces
and in linear regression, approximating dependent variables
becomes harder as the number of independent variables decreases. Indeed, linear alignment works worse when we
align from a representation space with lower dimensionality
to a representation space with higher dimensionality.


Leveraging representation space alignment, specifically to
CLIP, we perform text-to-concept, where text descriptions
of semantic concepts are encoded as vectors that can be directly compared (i.e. via cosine similarity) with the aligned
features of images obtained from an off-the-shelf vision
encoder (see Figure 1). Despite its simplicity, alignmentbased text-to-concept is surprisingly effective, which, after
further detailing our method, we demonstrate qualitatively
and quantitatively in this section. Notably, we show that the
similarities of aligned image representations to class vectors
obtained via text-to-concept enables _zero-shot classification_
_for non-CLIP models off-the-shelf_, with zero-shot accuracy
of much simpler models at times exceeding that of CLIP.


We define text-to-concept as a procedure for obtaining vectors corresponding to concepts described as text that can be
directly compared (i.e. via cosine similarity) to image representations from a fixed vision encoder. Our method begins
with a string describing some concept, like “red food”. We
then prepend this string with a number of template prompts
(e.g. “a photo of _{}_ ”); we use the same template prompts
as in CLIP’s original paper for ImageNet zero-shot classification. Then, we embed the templated text to CLIP space
using CLIP’s text encoder, and average the resultant vectors
over all templates to obtain a single concept vector (as is
standard). For some object agnostic concepts, such as contexts like “in a tree”, we can encode a general prompt like
“a photo of an object in a tree”, or we can obtain a more
refined vector by encoding “ _{prompt} {class name}_ in a
tree”, averaging over all choices for _class name_ and _prompt_ .
There are countless ways to prompt engineer; we elect to
use general prompts in most cases, as prompt engineering is
not the focus of our work. [1]
Then, for a given model, we train a linear layer to align
its representation space to CLIP; specifically, we use CLIP
ViT-B/16. We pass ImageNet training images to the given
model’s feature encoder and CLIP’s vision encoder, resulting in a dataset of paired representations with which we train
our aligner (Section 3). Now, we have two functions that
1See Appendix E for complete details on all prompts used.


_Figure 4._ Text-to-concept can encode finer-grain concepts, like combinations of concepts ( _“red food”_ ) or textures ( _“polka-dotted”_ ).
map to CLIP’s vision space: the CLIP text encoder (since
the text and vision representation spaces are shared), and the
composition of the given model’s encoder with the linear
aligner. Since the concept vector obtained via CLIP’s text
encoder and aligned representations from the given model
are both mapped to the same space, we can compare them
directly, thus satisfying our definition of text-to-concept. Alternatively, we could train an aligner from CLIP to the given
model’s representation space, and align the text embedding
instead of the features. We found this method to be less effective, possibly because the dimensionality of CLIP space
is lower than most models we study. Since our aligner is a
simple affine transformation, alignment minimally changes
the content of the representation obtained from the off-theshelf model. Also, note the efficiency of our approach: after
training a linear layer once, we can encode any number of
new concepts from text at no additional training cost.
**Qualitative Validation** : Figures 2 and 4 show images selected based on the cosine similarity of their aligned representations (obtained using off-the-shelf encoders and trained
linear aligners) to certain concept vectors. For each concept,
we present the classes with the highest average similarity, as
well as the most and least similar images within them. The
retrieved classes are sensible for each concept (e.g. _Amer-_
_ican Lobster_ for “red food”). Sorting images within each
class separates examples where the concept is extremely
prominent from those where the concept is absent (e.g. images of uncooked lobsters are least similar to the “red food”
concept). Note that the models used to obtain the image
representations differ in architecture, training objective and
supervision from CLIP, and most notably, they have not been
trained with any text/concept supervisions. Thus, it is surprising that we can easily connect these visual concept representations to the CLIP text embeddings. Nonetheless, over
a range of concept types (a context, a combination of color
and a concept, and a texture), our visualizations qualitatively
validate our proposed text-to-concept approach. We now
turn to zero-shot classification for quantitative validation.


CLIP models perform zero-shot classification by comparing
image representations to embeddings of text strings describing each class: the predicted class is the one whose text
embedding is most similar to the test image’s representation.
This is referred to as zero-shot since no labeled instances
from the candidate classes are used. Considering classes
as concepts, we can then use text-to-concept to obtain vectors that are directly comparable to aligned representations
from off-the-shelf vision encoders, thus extending CLIP’s
zero-shot capabilities. The accuracy of zero-shot classification serves as a quantitative measure of the quality of
text-to-concept vectors. Indeed, when concept vectors align
better with representations of samples in the class, zero-shot
accuracy is higher. Thus, we explore zero-shot classification
over many datasets to shed insight on when and how well


_Figure 5._ The zero-shot capabilities of CLIP can extend to off-the-shelf vision encoders via alignment based text-to-concept. ( **Left** )
Models trained on ImageNet can recognize coarse categorizations of ImageNet classes, despite never explicitly being taught them. ( **Right** )
Off-the-shelf models remain strong zero-shot classifiers even when images are out of distribution. In some cases, they surprisingly surpass
the accuracy of the CLIP vision encoder whose jointly-trained text encoder was used to embed each class vector.
text-to-concept works. We consider models over diverse
architectures and training procedures, though all models
are roughly equal in size ( _∼_ 25M parameters) and are only
trained on ImageNet (except for CLIP). Also, the baseline
CLIP model (ViT-B/16) whose text encoder is used to embed concepts is much larger in size ( _∼_ 80M parameters); this
baseline is intended more so as an upper bound.
First, we ask if models can recognize new categorizations
of the data they were trained over. Namely, we consider
coarse grained categorizations of ImageNet classes (e.g. distinguishing _insects_ from _carnivores_, see (Xiao et al., 2020;
Santurkar et al., 2020) and Appendix F). We also investigate
if these coarse grained concepts can still be recognized as
image data is taken out of distribution. Figure 5 displays the
results. We observe impressive zero-shot performance in
both cases. For example, on a 17-way classification problem,
self-supervised ViTs achieve 85% accuracy, despite never
receiving supervision about these classes, or any classes at
that. Shockingly, in a few cases, even the performance of the
CLIP model whose text encoder (with which it was jointly
trained) was used to obtain concept vectors is surpassed.
To stress test text-to-concept, we consider tasks that require
models to recognize characters (specifically digits) or primitive concepts, like textures, colors, and shapes. We observe
most models only marginally surpass random accuracy for
character recognition tasks. Oddly, the adversarially trained
ResNet is roughly twice as good as other models in zeroshot MNIST classification, though it still performs far worse
than the baseline CLIP model, which also struggles. This
suggests models simply may not have any notion as to what
_Figure 6._ Edge cases for zero-shot classification. ( **Left** ) Models
struggle with OCR. ( **Right** ) Models can recognize some primitive
concepts by name. Same legend as figure 5.
distinguishes digits from one another, which is not surprising given that it would not be very useful for understanding
ImageNet images. On the other hand, models achieve far
better than random performance in recognizing primitive
concepts, which appear in ImageNet as low level features
for more abstract notions. Interestingly, color recognition
is a task where most models outperform the CLIP baseline,
suggesting the CLIP ViT may have reduced color sensitivity
relative to other primitive concepts.
While these experiments validate our proposed text-toconcept method, it is also remarkable that these off-the-shelf
models, who have much smaller training sets (roughly 0 _._ 3%
the size of CLIP’s) and receive far less supervision, are
comparable to CLIP in recognizing the unseen classes we
consider. This suggests that models learn far more than


what they are taught. In other words, models discover
many semantic concepts and organize their representation
spaces so that these concepts are roughly linearly separable,
even when they are only explicitly directed to separate 1000
classes or to simply draw representations of similar inputs
close to another. Thus, even models trained with elementary techniques likely contain much richer representation
spaces than their use case requires. The success of transfer
learning supports this claim, as a small amount of labeled
data is sufficient for a model to recognize ‘new’ concepts,
implying they had some notion of the concepts before. Textto-concept can enable better understanding and utilization
of these rich representation spaces, _without requiring new_
_labeled samples_ .


_Figure 7._ Example inference for a Concept Bottleneck Model
(CBM) obtained via training a linear layer on zero-shot concepts.
Since logits in the CBM are linear functions of concept scores, we
can precisely quantify the contribution of concepts to each logit.
The zero-shot results suggest models are already aware of
many concepts beyond those which they are directly trained
to learn. One case where knowledge of concepts related to
the classification task is salient is Concept Bottleneck Models (CBMs) (Koh et al., 2020). CBMs are intepretable by
design, as they first predict the presence of concepts using a
black box, and then obtain class logits with a white box (e.g.
linear layer) atop concept predictions. Thus, the contribution of each concept to the predicted logit can be computed
directly, allowing predictions to be faithfully explained with
semantic reasons. A major barrier to using CBMs is that
they require concept supervision, which can be prohibitively
expensive. Text-to-concept, however, alleviates this constraint, thanks to zero-shot concept prediction.
We use RIVAL10 classification (Moayeri et al., 2022) as
an example for how a CBM can be implemented with _no_
_concept supervision_ using text-to-concept. RIVAL10 is
an attributed dataset, though we do not use these labels
during training. We use RIVAL10 because a linear classifier operating on ground truth attribute labels achieves
_Figure 8._ Concept similarities can reveal distribution shifts, like in
ObjectNet, where photos are taken within people’s homes.
94 _._ 5%, indicating that a CBM could be effective. Further,
the attribute labels allow for quantifying the quality of the
zero-shot concept vectors we obtain.
To implement the network, we use text-to-concept to encode the 28 attributes annotated in RIVAL10 as vectors
in CLIP space. We then compute the similarities between
the attribute vectors and aligned (to CLIP) features from
an ImageNet pretrained ResNet-50. Finally, we fit a linear layer atop image-attribute similarities (i.e. in representation space) to predict class labels. Note that the only
training we conduct is that of the final classification head
and of the aligner, both of which are linear layers, making
them time and sample efficient to optimize. The resultant
CBM achieves 93 _._ 8% accuracy, and yields the desired interpretability advantages, as shown in figure 7. Moreover,
using image-attribute similarities (via text-to-concept) as
a score for predicting attributes achieves an AUROC of
0 _._ 8, with 72% of attributes achieving at least an AUROC of
0 _._ 75. Thus, zero-shots concepts are relatively accurate in
predicting RIVAL10 attributes. See appendix G for details.

5.1. Concept-Bottleneck Networks for Free


The interpretability benefits of text-to-concept also apply to
demystifying large datasets. Specifically, one can discern
the presence of a concept in their data by using text-toconcept to obtain a corresponding vector, and computing the
similarity of this vector to all aligned images representations.
As modern datasets continue to grow, the need for efficient
concept-based summaries of these datasets will also grow;
text-to-concept can provide such summaries easily.
Moreoever, one can track the distribution of concept similarities for a stream of data over time. Suppose for example
a model is deployed to a new setting and it begins to fail.
By comparing the distribution of concept similarities in the
training set to the new data, one can diagnose the distribution
shifts at play. As a proof of concept, we inspect ObjectNet


(Barbu et al., 2019), a challenging distribution shift for ImageNet models consisting of images taken within people’s
homes. Figure 8 shows the distribution of similarities between the vector for the concept ‘indoors’ and aligned image
representations obtained from a ResNet-50 of ImageNet and
ObjectNet samples. For ObjectNet, the distribution is significantly (as determined with a Kolmogorov-Smirnov test)
shifted to the right compared to ImageNet. In practice, one
may maintain a bank of concepts and track similarities over
their stream of data, automatically flagging concepts which
experience significant shift.


_Figure 9._ Images retrieved based on similarity of their representation to multiple text-to-concept vectors. Retrieved images satisfy
the multiple conditions listed above each image. _∼_ denotes ‘not’.
Concept logic with text-to-concept enables searching over images,
while allowing the use of any vision encoder to represent them.
Text-to-concept enables the computation of the similarity of
a model’s representation for an image to an arbitrary concept. Given a corpus of data and a vision encoder trained to
represent said data, we can retrieve images using text, based
on their similarity to text-to-concept vectors. While one may
combine all keywords in a string to obtain a single composite concept vector, we observe suboptimal performance with
this approach, as words receive imbalanced attention and
negations are often ignored by CLIP’s text encoder.
A simple alternative is to retrieve images that satisfy a set of
conditions. For example, instead of searching for “a dog on
the beach at sunset”, we can separately encode the concepts
‘dog’, ‘on the beach’, and ‘at sunset’. We then filter images
based on their similarity for each concept; we use thresholds
based on the distribution of similarities for a given concept
(i.e. at least 3 standard deviations above the mean). Analogously, we can encode negative conditions by requiring similarity to be below some threshold. Figure 9 demonstrates
the effectiveness of our approach, retrieving rare images via
concept logic (details in appendix H). While concept logic
for image search can be done over CLIP vision embeddings,
the results may be suboptimal when querying over a specific dataset for which CLIP was not finetuned, particularly
compared to a vision model trained on that dataset.
Swin (S) ResNet-50 Dino ViTs8
94 _._ 48% 95 _._ 14% 92 _._ 18%
_Table 1._ Percent of captions for decoded class vectors deemed
relevant to images in the class by human annotators.


Text-to-concept grants insight into the representation spaces
of deep models by mapping semantic notions expressed as
words directly to concept vectors. However, humans still
need to conjecture what concepts may be relevant before
probing a representation space. We now ask, can we directly
map concept activation vectors to text? We refer to this as
_concept-to-text_, and propose an implementation using alignment to CLIP and generative language models (Figure 1).
Similar to how we that observe diverse vision models learn
to store information in similar ways, allowing for crossmodel alignment, we argue that language models and vision
models similarly learn much of the same information, and
can thus be plugged into one another with ease. Recent work
supports this claim, as vision models have been stitched to
generative language decoders to perform image captioning
and visual question answering (Merullo et al., 2022; Eichenberg et al., 2021). Notably, Mokady et al. (2021) captions
an image by feeding its CLIP image embedding through a
finetuned version of GPT-2 (Radford et al., 2019). A follow
up work, ZeroCap (Tewel et al., 2021), similarly decodes
with GPT-2 while receiving guidance from a CLIP embedding, but does so without requiring any tuning of either
CLIP or GPT-2. We elect this method as it further demonstrates how existing models can work together off-the-shelf.
However, other decoders could easily be put in place of
ZeroCap if desired. We highlight this flexibility as it entails
that concept-to-text will continue to improve as individual
components are improved (e.g. GPT-2 _→_ Chat-GPT).
With our linear aligners, we can already map representations from off-the-shelf encoders to CLIP. Thus, with no
additional training, we can perform elementary concept-totext by simply feeding aligned features to ZeroCap. While
ZeroCap expects CLIP embeddings of natural images as
input, we conjecture that passing a vector encoding some
semantic notion can similarly be decoded. To asses this
claim, we consider the task of decoding classification head
vectors. These vectors exist in the original model space, and
should encode information relevant to their corresponding
ImageNet class. Thus, we can quantify the effectiveness of
our elementary concept-to-text method by seeing if decoded
class vectors indeed describe the desired class. Specifically,
we use the prompt “Image of a ” and set the desired sequence length to 1 so that a single word is decoded per class
vector. We perform a human study to answer this question,


showing MTurk workers a collage of images from a given
class, along with the caption obtained from decoding the
class vector, and asking if this caption is relevant to the
images shown. The results, shown in Table 1, show that in
over 92% of cases, our naive approach to concept-to-text
appears effective in decoding class vectors for three diverse
models. See Appendix I for additional details.


This project was supported in part by Meta grant 23010098,
NSF CAREER AWARD 1942230, HR001119S0026
(GARD), ONR YIP award N00014-22-1-2271, Army Grant
No. W911NF2120076 and the NSF award CCF2212458.


Korchi, A. and Ghanou, Y. 2d geometric shapes dataset – for
machine learning and pattern recognition. _Data in Brief_,
32:106090, 07 2020. doi: 10.1016/j.dib.2020.106090.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers
of features from tiny images. 2009.
LeCun, Y., Cortes, C., and Burges, C. Mnist handwritten digit database. _ATT Labs [Online]. Available:_
_http://yann.lecun.com/exdb/mnist_, 2, 2010.
Lenc, K. and Vedaldi, A. Understanding image representations by measuring their equivariance and equivalence. In
_Proceedings of the IEEE conference on computer vision_
_and pattern recognition_, pp. 991–999, 2015.
Liu, Z., Luo, P., Wang, X., and Tang, X. Deep learning face
attributes in the wild. In _Proceedings of International_
_Conference on Computer Vision (ICCV)_, December 2015.
Merullo, J., Castricato, L., Eickhoff, C., and Pavlick, E. Linearly mapping from image to text space. _arXiv preprint_
_
Moayeri, M., Pope, P. E., Balaji, Y., and Feizi, S. A comprehensive study of image classification model sensitivity
to foregrounds, backgrounds, and visual attributes. _2022_
_IEEE/CVF Conference on Computer Vision and Pattern_
_Recognition (CVPR)_, pp. 19065–19075, 2022.
Mokady, R., Hertz, A., and Bermano, A. H. Clipcap: Clip
prefix for image captioning. _ArXiv_, abs/2111.09734,
2021.
Moschella, L., Maiorca, V., Fumero, M., Norelli, A., Locatello, F., and Rodola, E.` Relative representations
enable zero-shot latent space communication. _ArXiv_,
abs/2209.15430, 2022.
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu,
B., and Ng, A. Y. Reading digits in natural
images with unsupervised feature learning. In
_NIPS Workshop on Deep Learning and Unsuper-_
_vised Feature Learning 2011_, 2011. [URL http:](http://ufldl.stanford.edu/housenumbers/nips2011_housenumbers.pdf)
[//ufldl.stanford.edu/housenumbers/](http://ufldl.stanford.edu/housenumbers/nips2011_housenumbers.pdf)
[nips2011_housenumbers.pdf.](http://ufldl.stanford.edu/housenumbers/nips2011_housenumbers.pdf)
Oikarinen, T. and Weng, T.-W. Clip-dissect: Automatic
description of neuron representations in deep vision networks. _arXiv preprint
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J.,
Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga,
L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison,
M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L.,
Bai, J., and Chintala, S. Pytorch: An imperative style,
high-performance deep learning library. In _Advances_
_in Neural Information Processing Systems 32_, pp. 8024–
8035. Curran Associates, Inc., 2019.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and
Sutskever, I. Language models are unsupervised multitask
learners. 2019.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G.,
Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark,
J., Krueger, G., and Sutskever, I. Learning transferable
visual models from natural language supervision. In
_ICML_, 2021.
Salman, H., Ilyas, A., Engstrom, L., Kapoor, A., and Madry,
A. Do adversarially robust imagenet models transfer better? _Advances in Neural Information Processing Systems_,
33:3533–3545, 2020.
Santurkar, S., Tsipras, D., and Madry, A. BREEDS: benchmarks for subpopulation shift. _CoRR_, abs/2008.04859,
2020. [URL https://arxiv.org/abs/2008.](https://arxiv.org/abs/2008.04859)
[04859.](https://arxiv.org/abs/2008.04859)
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C.,
Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis,
C., Wortsman, M., et al. Laion-5b: An open large-scale
dataset for training next generation image-text models.
_arXiv preprint
Tewel, Y., Shalev, Y., Schwartz, I., and Wolf, L. Zerocap:
Zero-shot image-to-text generation for visual-semantic
arithmetic. _2022 IEEE/CVF Conference on Computer Vi-_
_sion and Pattern Recognition (CVPR)_, pp. 17897–17907,
2021.
Thomee, B., Shamma, D. A., Friedland, G., Elizalde, B., Ni,
K., Poland, D., Borth, D., and Li, L.-J. Yfcc100m: The
new data in multimedia research. _Communications of the_
_ACM_, 59(2):64–73, 2016.
Tsimpoukelli, M., Menick, J., Cabi, S., Eslami, S. M. A.,
Vinyals, O., and Hill, F. Multimodal few-shot learning
with frozen language models. In _Neural Information_
_Processing Systems_, 2021.
[Wightman, R. Pytorch image models. https://github.](https://github.com/rwightman/pytorch-image-models)
[com/rwightman/pytorch-image-models,](https://github.com/rwightman/pytorch-image-models)
2019.
Wold, S., Esbensen, K., and Geladi, P. Principal component analysis. _Chemometrics and intelligent laboratory_
_systems_, 2(1-3):37–52, 1987.
Xiao, H., Rasul, K., and Vollgraf, R. Fashion-mnist: a
novel image dataset for benchmarking machine learning
algorithms. _ArXiv_, abs/1708.07747, 2017.
Xiao, K., Engstrom, L., Ilyas, A., and Madry, A. Noise or
signal: The role of image backgrounds in object recognition. _ArXiv preprint


Zhai, X., Wang, X., Mustafa, B., Steiner, A., Keysers, D.,
Kolesnikov, A., and Beyer, L. Lit: Zero-shot transfer with
locked-image text tuning. _2022 IEEE/CVF Conference_
_on Computer Vision and Pattern Recognition (CVPR)_, pp.
18102–18112, 2021.
Zhang, R., Madumal, P., Miller, T., Ehinger, K. A., and Rubinstein, B. I. P. Invertible concept-based explanations for
cnn models with non-negative concept activation vectors.
In _AAAI Conference on Artificial Intelligence_, 2020.
Zhu, J.-Y., Park, T., Isola, P., and Efros, A. A. Unpaired
image-to-image translation using cycle-consistent adversarial networks. _2017 IEEE International Conference on_
_Computer Vision (ICCV)_, pp. 2242–2251, 2017.


Sup
ResNets
Robust
ResNets
Sup
ViTs
SS
ResNets
SS
ViTs
CLIPs
1.00
0.75
0.50
0.25
Sup
ResNets
Robust
ResNets
Sup
ViTs
SS
ResNets
SS
ViTs
CLIPs
1.00
0.75
0.50
0.25
|0.98|1.03|0.82|0.86|0.79|0.80|
|---|---|---|---|---|---|
|0.87|0.98|0.76|0.80|0.73|0.73|
|1.01|1.08|0.96|0.93|0.86|0.68|
|0.88|0.94|0.77|0.90|0.78|0.79|
|0.93|0.97|0.83|0.81|0.88|0.82|
|0.72|0.76|0.58|0.55|0.46|0.80|
|0.77|0.59|0.35|0.47|0.77|0.67|
|---|---|---|---|---|---|
|0.59|0.79|0.36|0.53|0.79|0.68|
|0.38|0.36|0.54|0.34|0.69|0.60|
|0.60|0.65|0.42|0.76|0.85|0.73|
|0.54|0.56|0.38|0.51|0.87|0.71|
|0.42|0.43|0.23|0.39|0.72|0.77|
Robust
ResNets
Sup
ViTs
SS
ResNets
SS
ViTs
CLIPs
Sup
ResNets
Robust
ResNets
Sup
ViTs
SS
ResNets
SS
ViTs
CLIPs
Sup
ResNets
Target
(a) The average of retained accuracy.
Target
(b) The average of _R_ [2] .
_Figure 10._ **(Left)** shows the average of retained accuracy. More precisely, the value in row _r_ and column _c_ is the average of retained
accuracy when doing alignment from representation space of model _s_ to that of model _t_ where _s_ is a model in group _r_ and _t_ is a model of
group _c_ . **(Right)** shows the average of _R_ [2], i.e., same as above, value in row _r_ and column _c_ is the average of _R_ [2] in linear alignment from
models in group _r_ to models of group _c_ . Note that “Sup” stands for Supervised while “SS” stands for Self-Supervised training procedure.
Note that all models are pretrained on ImagenNet-1K except CLIPs. More details on models used here can be found in Section C.


In this section, we conduct an extensive set of experiments where we evaluate linear alignment between many pairs of
models (see Section C for more details on models). We also consider cases where two models are significantly different, i.e.,
they may have different architectures, or they are trained with different procedures (supervised learning, self-supervised
learning, etc).
we formally define _aligned accuracy_, i.e., the accuracy of alignment on _D_ test, and _retained accuracy_, i.e., the ratio of aligned
accuracy to the target model accuracy, when linear alignment is done from the representation space of _fs_ to that of _ft_ . Note
that _W, b_ are the solutions of optimization problem provided in Section 3.
aligned accuracy :=
_D_ test
_|_ _|_
 - 1 [ _gt_ ( _hW,b_ ( _x_ )) = _y_ ]
( _x,y_ ) _∈D_ test
aligned accuracy
retained accuracy := _|D_ 1test _|_ ~~�~~ ( _x,y_ ) _∈D_ test [1][ [] _[g][t]_ [ (] _[f][t]_ [(] _[x]_ [)) =] _[ y]_ []] _[.]_
As seen in Figure 10, linear alignment generally works well in the sense of both _R_ [2] and retained accuracy. In many cases,
we see _R_ [2] score above 0 _._ 6, which indicates that a significant portion of variance is captured in the linear regression. We also
see better values for _R_ [2] on the diagonal of right heatmap in Figure 10 as representation spaces of models with the same
architecture or training procedure are more linearly transformable. Furthermore, models are surprisingly capable of retaining
the accuracy of the target models when a linear transformation is done on their representation spaces. Some models are
more capable of retaining accuracy, e.g., as seen in Figure 10, _S_ upervised Vision Transformers get even better accuracy
when their feature spaces are **linearly transformed** into other models’ feature spaces and the **classification heads of the**
**other models** are applied. Figure 11 shows the average of retained accuracy for each individual source model _s_ along the
average of retained accuracy for each group of source models.
This is due to the fact that richer and more informative representation spaces make classification easier. Note that in our
experiments, all models except CLIPs are trained on ImageNet-1K dataset (Deng et al., 2009) but we evaluate all of them
with ImageNet. Note that this dataset is remarkably hard and big, which further supports the idea that linear alignment is


_Figure 11._ For each model _s_, average of retained accuracy when doing alignment from model _s_ to all other models is reported.
possible and according to Figure 10, different models are actually learning linearly transformable concepts.


Setting _D_ train := _{xi}i_ _[N]_ =1 [and] _[ D]_ [test][ :=] _[ {][x][i][}]_ _i_ _[N]_ =1 _[′]_ [, we note that the optimization problem to obtain] _[ W]_ [ and] _[ b]_ [ is]
_N_
_i_ =1
- _W_ _[T]_ _fs_ ( _x_ [train] _i_ ) + _b_ _ft_ ( _x_ [train] _i_ )�2 _._ (1)
_−_
_W, b_ = arg min
_W,b_
_N_
With a proper set of hyperparameters around 6 epochs are enough to converge to the optimal solution. However, re-scaling
representation spaces of models so that the variance of elements in the space becomes constant, is crucial (See Section D).
This is due to the fact that some models embed inputs into very low variance spaces, which degrades the performance of
linear alignment due to precision in computations.
Additionally, we take into account the optimization problem given in (A.1) and consider the effect of the number of images
that we involve in optimizing (A.1). We observe that using only a random subset of the training set of ImageNet is sufficient
to find _W_ and _b_, as seen in Figure 12, 1 _/_ 5 of ImageNet training samples is roughly enough to retrieve the target model
accuracy. if we use only images of some particular classes to optimize (A.1), we can retrieve the target accuracy by just
using around 1 _/_ 3 of ImageNet classes.


We now present alternate objectives to enable alignment, specifically to vision language models like CLIP. The method
we present optimizes a linear layer aligner for the regression task of predicting features for one model given features from
another. An alternate approach is to optimize the aligner _directly for classification_ . That is, we use a cross entropy loss
_ℓ_, obtaining logits for a sample _x_ by passing features _f_ ( _x_ ) from a fixed encoder _f_ through a trainable linear aligner with
parameters _W, b_ before finally having their cos-sine similarity taken with a set of text embeddings for each class obtained
via CLIP’s text encoder. To recap, we solve min _W,b_ - _x,y∈D_ _[ℓ]_ [(][sims][(] _[x]_ [)] _[, y]_ [)][ for a dataset] _[ D]_ [ of labeled images (x,y), where]
sims( _x_ ) _i_ = cossim( _W_ _[T]_ _f_ ( _x_ ) + _b, ti_ ), where _ti_ is the CLIP text embedding of the name of class _yi_ . Essentially, we fix a
classification head in CLIP space as the text embeddings of the class names for a task. Then, we optimize a layer stitching a
|Col1|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|Col10|Col11|Col12|
|---|---|---|---|---|---|---|---|---|---|---|---|
|||||||||||||
|||||||||||||
|||||||||||||
|||||||||||||
|||||||M
D
|odel
ino ViTs
oCo Re|8
 Net50||||
|||||||
R
T|
esNet50
arget Mo|
 del (Sim|CLR Re|sNet50X|1)|
100 200 300 400 500 600 700 800 900 1000
Number of Classes

A.2. Alternate Objectives for Alignment to CLIP


|Col1|Col2|Col3|Col4|Col5|Col6|Col7|Col8|
|---|---|---|---|---|---|---|---|
|||||||||
|||||||||
|||||||||
|||||||||
|||||||||
|||||~~Mod~~
Dino
~~MoC~~|~~el~~
 ViTs8
~~o ResNet50~~|||
|||||
ResN
Targ|
et50
et Model (SimC|LR ResNet50X|1)|
10 [4] 10 [5] 2 _×_ 10 [5] 3 _×_ 10 [5] 4 _×_ 10 [5] 5 _×_ 10 [5]
Number of Samples
_Figure 12._ **(Left)** shows the aligned accuracy when linear transformation is only optimized on images with particular labels. We randomly
select labels and increase the number of labels(classes) to see how retained accuracy changes. while **(Right)** shows the aligned accuracy
when linear alignment is solved on a random subset of images. Alignment is done from three different models to SimCLR ResNet50X1.
We observe that all training images are not necessary to have a reliable alignment. In other words, aligned accuracy can reach to its
maximum by only considering small portion of images or classes.
Overall ResNets ViTs
Related to ImageNet -4.99% -1.50% -8.47%
Not Related to ImageNet 11.21% 14.70% 7.72%
_Table 2._ Average gain in zero-shot accuracy of the original aligner optimization (i.e. regression on image features) over the alternate (i.e.
cross entropy loss using fixed classification head). Two ResNet50s (standard, MoCo) and two ViTs (DeiT, MoCo) considered. Datasets
denoted as “Related to ImageNet” are: ImageNet9 and all BREEDS datasets (direct coarse grained categorizations of ImageNet images),
and CIFAR10, STL10, and Fashion MNIST (ImageNet classes/coarse categories in OOD data). Datasets denoted as “Not Related to
ImageNet” are: CelebA hair classification, DTD, Colors, Shapes, SVHN.
given image encoder to this classification head.
The requirement of labeled data is one drawback of this baseline compared to our method of aligning representations.
Secondly, this training does not necessarily entail that the aligner truly learns to map to the target representation space, as
any information irrelevant to the selected class vectors may be lost. Thus, while zero-shot performance on tasks similar
to the one optimized for will likely exceed our method, it may come at the cost of significantly reduced performance on
unrelated tasks. Indeed, we find this to be the case: when optimizing the aligner for ImageNet classification, we observe the
alternate method to perform roughly 11% worse on zero-shot classification tasks unrelated to ImageNet, though performance
on ImageNet-like tasks is roughly 5% better (see table 2).
We note that the ViTs we consider benefit much more from the alternate method than ResNets. We conjecture this is due to
the lower dimensionality of ViT feature space relative to CLIP, which makes our linear regression task under-determined and
thus challenging. In contrast, the alternate method provides a less stringent optimization task, since the aligner needs only to
map representations of samples from different classes to separable clusters in CLIP space. This makes the training easier
and more successful for ViTs, though it comes at two key limitations: the baseline requires labeled data, and the alignment
is less reliable for concepts that are not directly related to the classification task the aligner is optimized for.
Finally, it may be possible to solve for alignment in a completely data-free manner. Namely, one may seek to solve a matrix
equation to re-express an existing classification head as the product of a learnable aligner and a fixed classification head
in CLIP space (e.g. obtained by embedding the names of the classes for the task). In addition to being data-free, this
method potentially would result in faithful preservation of the original model’s behavior, while still unlocking all of the
interpretability benefits of text-to-concept. Given the importance of faithfulness in interpretability, and the flexibility of not
requiring data, we believe solving alignment in this manner may be a promising line of future investigation.


1 _._ 0
0 _._ 9
0 _._ 8
|Col1|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|Col10|
|---|---|---|---|---|---|---|---|---|---|
|||||||||||
|||||||||||
|||||||||||
||Mod|el||||||||
||CLIP
SS R
SS V|OpenAI
 esNets
 iTs||||||||
||~~Sup~~
Sup|~~ ResNets~~
 ViTs||||||||
||Rand|om Matri|x|x||||||
0 5 10 15 20 25 30 35
_i_
|0|Col2|Col3|5|Col5|Col6|10|Col8|Col9|15|Col11|Col12|20|Col14|Col15|25|Col17|Col18|30|Col20|Col21|35|Col23|Col24|Col25|
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
||||||||||||||||||||||||||
||||||||||||||||||||||||||
||||||||||||||||||||||||||
||||||||||||||||||||||||||
||||||||||||||||||||||||||
||||||||||||||||||||||||||
||||||||||||||||||||||||||
||||||||||||||||||||||||||
||||||||||||||||||||||||||
||||||||||||||||||||||||||
||||||||||||||||||||||||||
||||||||||||||||||||||||||
||||||||||||||||||||||||||
||||||||||||||||||||||||||
||||||||||||||||||||||||||
||||||||||||||||||||||||||
||||||||||||||||||||||||||
||||||||||||||||||||||||||
||||||||||||||||||||||||||
||||||||||||||||||||||||||
||||||||||||||||||||||||||
_Figure 13._ Shows the heatmap of the average of matrix
_W_ _[T]_, when doing alignment between every pair of 4
CLIP models pre-trained on OpenAI dataset. Two of
these 4 models use ResNet architecture while two others
use Vision Transformers. We observer that the matrix is
almost diagonal, which implies that there is a 1-1 relation
between top principal components in CLIP models.


0 _._ 7
0 _._ 6
0 _._ 5
0 _._ 4
0 _._ 3
0 _._ 2
_Figure 14._ Shows the plot of the average of diag _i_ when we do linear alignment between all pairs of models within each group of
them. Generally, we see larger values for diag _i_ when _i_ is smaller.
This implies that 1-1 correspondence holds more for top principal components. High values of diag _i_ show that the geometry of
principal component space of top components within CLIP models,
supervised ResNets, and self-supervised ResNets are approximately
same. For more details regarding models, see Section C.
In this section, we extend linear alignment to the space of principal components. We know that the effective dimension
of representation spaces of models is relatively low. This is mostly due to the existence of redundant information in these
spaces so that many of the features can be approximated with a linear combination of others or the existence of some
noise features. As a result, it is reasonable to consider the representation spaces in a more abstract manner. To do so, we
use _Principal Component Analysis_ (PCA) (Wold et al., 1987). Let pc0 _,_ pc1 _, ...,_ pc _d−_ 1 be the principal components of the
representation space of model _m_, i.e., _fm_ ( _x_ [train] _i_ ) _i_ =1 [, in decreasing order of their corresponding eigenvalues][ 2][ . We take]
_{_ _}_ _[N]_
top _k_ principal components and project representations into the space of these components. For each point _fm_ ( _x_ ), we get a
vector _qm_ ( _x_ ) R _[k]_ where
_∈_
_qm_ ( _x_ ) := ( _fm_ ( _x_ ) _·_ pc _i_ ) _i_ _[k]_ =0 _[−]_ [1] _[.]_
Now, for each input image _x_, we have a low-dimension embedding which due to the properties of PCA, retains a significant
portion of the information of the original representation _fm_ ( _x_ ). Like Section 3, we apply linear alignment on **principal**
**components space** of model _s_ to get that of model _t_ . Formally, we define linear transformation _h_ ( _z_ ) as
_h_ ( _z_ ) := _W_ _[T]_ _z_ + _b._
To find best _W_ and _b_, we follow (1) but replace _f_ with _q_ . Note that PC alignment, matrix _W_ and vector _b_ have significantly
lower dimensions, i.e., _W ∈_ R _[k][×][k]_ and _b ∈_ R _[k]_ .
Interestingly, we see that between many pairs of models, even though there is a significant difference in architecture,
there approximately exists a 1-1 correspondence between principal components. Indeed, _i_ -th element of _qt_ ( _x_ ) can be
approximated by _i_ -th element of _qs_ ( _x_ ). This implies that surprisingly, (1) **top principal components represent the same**
**abstract knowledge** even in different models, and (2) the **order of these top components is preserved** among models. A
visualization of matrix _W_ _[T]_ where each row is normalized is depicted in Figure 13. We measure the observation of 1-1
correspondence in a quantitative manner. Note that according to linear alignment,
_qt_ ( _x_ ) _i_
_≈_
_k_
- _Wi,j_ _[T]_ _[q][s]_ [(] _[x]_ [)] _[j][,]_
_j_ =1
2Note that before doing PCA, we centralize our points in representation space such that mean of points become _⃗_ 0.


where _qs_ ( _x_ ) _j_ denotes the _j_ -th element of _qs_ ( _x_ ) and _qt_ ( _x_ ) _i_ denotes the _i_ -th element of _qt_ ( _x_ ). We normalize each row in
_W_ _[T]_ and then measure how much elements close to the diagonal contribute in the approximation of _i_ -th element, i.e., for
each _i ∈{_ 0 _,_ 1 _, ..., k −_ 1 _}_, we define diag _i_ as
diag _i_ :=
min( _k−_ 1 _,i_ + _p_ )
_j_ =max(0 _,i−p_ )
- _Wi,j_ _[T]_ �2 _,_
where _p_ = 5 in our experiments. Also we take _k_ = 40 top principal components. Figure 14, shows the average value of
(diag _i_ ) _[k]_ _i_ =0 _[−]_ [1] [when doing alignment between every pair of models in different groups. We consider][ 5][ groups of models (see]
Section C for details about each group) with different architecture or training procedure. As observed in Figure 14, 1-1
correspondence holds more within CLIP models, supervised, and self-supervised ResNets while it doesn’t hold within
supervised vision transformers.


In this paper we have considered several different models in different categories (Radford et al., 2021), (Chen* et al., 2021),
(Caron et al., 2021), (He et al., 2016), (Salman et al., 2020), (Radford et al., 2021). All these models except CLIP models
[are trained on ImageNet-1K (Deng et al., 2009). For almost all of these models, pretraiend weights are obtained from timm](https://github.com/rwightman/pytorch-image-models)
library (Wightman, 2019) and (Ilharco et al., 2021). Our models are categorized in following groups.
  - **Supervised ResNets** include ResNet50, and ResNet18.
  - **Robust ResNets** include Robust Resnet50 _ℓ_ 2 _, ϵ_ = 0 _._ 25, Robust Resnet50 _ℓ_ 2 _, ϵ_ = 1 _._ 0, and Robust Resnet50 _ℓ_ 2 _, ϵ_ = 3 _._ 0.
  - **Supervised Vision Transormers** include Swin, Deit, and Convit models.
**–** Swin with patch size of 4 and window size of 7 includes Swin Small (S) and Swin Tiny (T).
**–** Deit with patch size of 16 includes Deit Small (S) and Deit Tiny (T).
**–** Convit includes Convit Small (S) and Convit Tiny (T).
  - **Self-Supervised ResNets** include MoCo ResNet50, Dino ResNet50, SimCLR ResNet50X1, and SimCLR
ResNet50X2.
  - **Self-Supervised Vision Transformers** include MoCo ViT base (B), MoCo ViT small (S), Dino ViTs 16, and
Dino ViTs 8, .
  - **CLIP** include
**–** CLIP ResNet101, CLIP ResNet50, CLIP ViT-B/16, and CLIP ViT-B/32 trained on OpenAI dataset.
**–** CLIP ResNet101 and CLIP ResNet50 trained on YFCC (Thomee et al., 2016).
**–** CLIP CLIP ViT-B/16 and CLIP CLIP ViT-B/32 trained on LAION (Schuhmann et al., 2022).


In terms of optimizing A.1, we use SGD optimizer and learning rate scheduler (implemented in Torch (Paszke et al., 2019))
with following hyperparameters:
optimizer = optim.SGD(lr=0.01, momentum=0.9, weight ~~d~~ ecay=5e-4)
scheduler = torch.optim.lr ~~s~~ cheduler.CosineAnnealingLR(T ~~m~~ ax=200)
We run optimization for 6 epochs. Note that before optimizing we re-scale representation spaces of models such that
variance of elements in matrix - _fs_ ( _x_ [train] 1 ) _, fs_ ( _x_ [train] 2 ) _, ..., fs_ ( _x_ [train] _N_ [)] - becomes 4 _._ 5.


_Figure 15._ Additional qualitative validation of text-to-concept. Observe that the samples least similar to the concept vector for “in snow”
break a common spurious correlation for their classes. Text-to-concept may then be used to identify challenging natural images within
datasets, towards mitigating spurious correlation dependencies.


When not otherwise specified, we use the default templates introduced in the original CLIP paper for ImageNet zero-shot
classification. They are as follows: _‘itap of a {}’, ‘a bad photo of the {}’, ‘a origami {}’, ‘a photo of the large {}’, ‘a {} in_
_a video game’, ‘art of the {}’, ‘a photo of the small {}’_ .
For Figure 2 and Figure 15, we append “in a tree” and “in snow” to the above templates, and also replace the ’ _{}_ ’ with names
for all ImageNet classes. The final concept vector is then an average of NUMBER OF TEMPLATES _×_ NUMBER OF CLASES
vectors. We do this because these correspond to contexts, which should be object agnostic. Similar results are obtained
without refinement (i.e. replacing _{}_ with ‘object’). We note that obtaining embedding text to CLIP’s space is very quick,
only taking seconds to encode a batch of thousands of short phrases.
Dataset Example Classes Prompt Citation
Coarse Grained Concepts In Distribution
IN9 dog, bird, wheeled vehicle a photo of _{}_ Xiao et al. (2020)
Living17 salamander, turtle, lizard a photo of a _{}_ Santurkar et al. (2020)
Nonliving26 bag, ball, boat a photo of a _{}_ Santurkar et al. (2020)
Entity13 garment, bird, reptile a photo of a _{}_ Santurkar et al. (2020)
Entity30 serpentes, passerine, saurian a photo of a _{}_ Santurkar et al. (2020)
Coarse Grained Concepts Out of Distribution
CIFAR10 airplane, automobile, bird a pixelated photo of a _{}_ Krizhevsky et al. (2009)
STL10 airplane, bird, car a photo of a _{}_ Coates et al. (2011)
Fashion MNIST T-shirt/top, Trouser, Pullover a black and white photo of _{}_ Xiao et al. (2017)
CelebA Hair brown hair, blonde hair a headshot of a person with _{}_ Liu et al. (2015)
Character Recognition
SVHN zero, one, two a photo of the digit “ _{}_ ” on a building Netzer et al. (2011)
MNIST zero, one, two a photo of the digit “ _{}_ ” LeCun et al. (2010)
Primitive Concepts
Textures banded, blotchy, braided a photo of something with _{}_ texture Cimpoi et al. (2014)
Color black, blue, brown a swatch of the color _{}_    Shape circle, octagon, square a diagram of the shape _{}_ Korchi & Ghanou (2020)
_Table 3._ List of datasets studied in Zero-Shot classification experiments (Section 4.2), along with example classes and the specific prompt
used. Note that we use an internal simple dataset for _Color_ .


_Figure 16._ Quality of using similarity to text-to-concept vectors for predicting RIVAL10 attributes. AUROC shown per attribute. Attributes
corresponding to parts are predicting more reliably. Over 70% of attributes achieve an AUROC of at least 0 _._ 75.
_Figure 17._ Extra examples of inference using the concept-bottleneck model (enabling direct measurement of each concept’s contribution
to a class logit, as shown) built atop a fixed vision encoder and text-to-concept. We include a misclassification in the rightmost panel.


We carry out zero-shot experiments over many datasets. We use slightly different prompts for each task, though we stress
that we did not optimize prompt engineering to obtain better results. All evaluated models use the same prompts. Table 3
shows details for prompts used, as well as example classes for each dataset, to give an idea as to what kind of text is used to
generate concept vectors. We refer readers to the original sources for more details on the datasets studied.
For color recognition, we construct a simple dataset that consists of one sample per the following classes: _black, blue, brown,_
_gray, green, orange, pink, purple, red, white, yellow_ . The sample in each class is a monocolor patch, with every pixel set to
have the color given by the class name. Also, for shape recognition, we use a subset of the shapes in the original dataset.
Specifically, we include the following shapes: _circle, octagon, square, star, triangle_ .


RIVAL10 (Moayeri et al., 2022) is a ten class classification problem operating on ImageNet images. Each image is also
annotated with 28 attributes, though we only use these to evaluate our associated text-to-concept vectors. It is analagous to
the classes in CIFAR10. Training the CBM is quite simple:
1. We obtain features for each image using a pretrained ResNet50.
2. We obtain vectors corresponding to each RIVAL10 attribute. We use standard templates with no class averaging
(simplest mode, no prompt engineering).
3. We use our trained aligner from ResNet50 to CLIP ViT-B/16 (section 3) to align features to CLIP space. Recall this is
just feeding saved features to a linear layer.


4. We compute and save the cosine similarity of each aligned image representation to the attribute concept vectors obtained
in step 2. Again, this amounts to normalizing two matrices before multiplying them together.
5. We train a linear layer mapping the similarities (obtained in previous step) of aligned representations and attribute
concept vectors to RIVAL10 class labels. We train this linear layer for 40 epochs with SGD.
One can see that the conversion of an existing encoder to the CBM for the desired task is simple and requires minimal
training (we only train a linear layer for the aligner and one for the classification head). We achieve 93 _._ 8% accuracy, and
AUROCs for each attribute prediction as shown in figure 16. Note that text-to-concept vectors for color attributes seemed to
be the least reliably, while vectors for part attributes are the best, achieving near perfect AUROC.
Given a vector of concept predictions _c_ and classification head vector corresponding to a class _w_, we compute the contribution
of concept _i_ to the class as ~~�~~ _wi×ci_
_j_ _[|][w][j]_ _[×][c][j]_ _[|]_ [. We call these logit shares, as each score is the share of activation on a given logit]
from one concept divided by the sum of contributions to that logit over all concepts.
Figure 7 and 17 shows how inference on a CBM is more interpretable, thanks to direct computation of logit shares with
respect to a set of concepts. Specifically, we list the predicted class and the three concepts with largest logit shares for the
predicted class. We additionally list the runner-up class, along with the concepts with the largest difference in activation
shares for the predicted logit and the runner up logit. These concepts amount to ones that are much more influential for the
predicted class than the runner up, and can provide insight as to why the predicted class was chosen over the runner up.
This interpretability benefit can be particularly useful in understanding failure modes. For example, the rightmost panel in
Figure 17 displays a _truck_ image misclassified as a _ship_ . The concept with the largest drop in logit share between the _ship_
and _truck_ logits is the color _blue_ . Arguably, this reveals a potentially problematic spurious correlation the CBM attributes to
the _ship_ class, perhaps because of the blue water and blue sky that is often present in images of ships. Because the color
blue is prominent in the truck image (from the sky), the model misclassifies it as _ship_ . Identifying spurious correlations is
the first towards improving the robustness of models to changes in spurious features, and the CBM makes identifying such
correlations (namely to human interpretable concepts) easier.


_Figure 18._ Overview of our concept logic image retrieval technique. Instead of directly searching for images of _orange cats that are not_
_indoors_, we decompose the query into concept constraints of varying degree. Our method grants the querier more control in their search,
and also circumvents limitations of CLIP’s text encoder when processing negations and longer queries.
Concept Logic simply applies a set of filters to obtain images that meet several concept requirements. The constraints
are of the form (concept, scale _k_, sign _s_ ). This means that the similarity of the aligned representation of an image to the
text-to-concept vector must either be _k_ standard deviations above or below (depending on _s_ ) the mean similarity for that
vector. The scale parameter allows for easy control of how strongly each concept in the query be should take into account.
Figure 18 diagrams our approach, specifically for the example of retrieving images of _orange cats that are not indoors_ . For
a set of concept constraints, we first encode them as input triples, indicating the severity with which each concept constraint


Encoder Concept Constraints All Retrieved Images
Dino-ResNet50
Dino-ResNet50
Standard ResNet50
(‘a dog’, 2.25, 1)
(‘the beach’, 2, 1)
(‘the sunset’, 2, 1)
(‘skis’, 4, 1)
(‘snow’, 2, -1)
(‘human’, 1,-1)
(‘cat’, 3, 1)
(‘orange’, 2, 1)
(‘indoors’, 0, -1)
_Table 4._ Complete details for Figure 9, including concept constraints in the form used to query images and all retrieved images for a set of
concept constraints and a fixed vision encoder.
should be applied. Reducing the severity of a constraint leads to the retrieval of more images, at the cost of including more
erroneous instances. The sign of the constraint allows for negation of certain concepts (e.g. returning images of cats that are
_not_ indoors). We observe CLIP’s text encoder to struggle with negations; concept logic circumvents this issue by returning
instances _least_ similar to a concept.
For each concept constraint, we encode the concept and compute the similarity of all aligned representations from the data
pool of interest to the concept vector, and then obtain a subset of images that satisfy the given constraint. Finally, we take
the intersection of all returned subsets, resulting in a final set of retrieved images that satisfy all concept constraints.
Table 4 outlines the complete details for the concept logic based image retrieval previewed in Figure 9. In the figure, we
present one of the retrieved images given a set of concept constraints and a fixed vision encoder. The table contains the
exact concept constraints, as well as all images that the query returns.


We now provide additional details on the concept-to-text experiment of Section 6. The method to perform concept-to-text
immediately follows from our work aligning diverse model spaces to CLIP, and the work of Tewel et al. (2021) (ZeroCap),


_Figure 19._ Sample HIT shown to MTurk workers. The decoded caption is ‘owl’, and the corresponding class name is ‘grey owl’.
which decodes CLIP vectors to text with no training required. Our use case is novel compared to those explored in ZeroCap,
as we decode _general_ vectors in a feature space, as opposed to representations directly corresponding to natural images.
Namely, we decode _classification head vectors_ from three pretrained ImageNet classifiers (Swin transformer, standard
supervised ResNet-50, and Dino-trained ViT) of roughly equivalent size. Recall that the predicted class of an image is the
argmax of the product of classification head vectors and the image’s representation (plus a bias vector). Thus, we would
expect each classification vector to correspond to a concepts relevant to the class, such as the class object itself, similar
object, or things that frequently co-occur with the class object.
We make one small modification to classification head vectors before decoding them. While they exist in the same space as
image representations, their norm conceivably may be much different than the norms of image representations, on which our
aligners are trained. So long as all classification head vectors have similar norm, the argmax of the product of classification
head vectors and an image representation would still be effective in predicting said image’s class. However, this would
result in poor alignment, as the classification head vectors may be out of distribution for the linear aligner. The solution to
this problem is simple: we rescale all classification head vectors by a constant so that the variance over all of their elements
is equal to the variance over all representations of images used to train the aligner. This minor modification drastically
improved performance, particularly for the transformer-based models we studied.
To recap, our procedure to decode a classification head vector is to (i) rescale it, (ii) align to CLIP using a linear aligner,
and (iii) decode using ZeroCap, with the prompt of “Image of a ” and a target output sequence length of 1. We choose a
sequence length of 1 so to reduce hallucinations from GPT-2. All other hyperparameters for ZeroCap are set to the default
captioning settings). We note this process is extremely efficient: in minutes, we can decode all one thousand classification
head vectors for a given model.
To assess the quality of our decodings, we use a human study conducted on Amazon Mechanical Turk. Figure 19 displays a
screenshot for a single task, where a human compares a collage of images from a given class to the single word decoded for the
corresponding classification head vector. Specifically, for the question ‘does the word _{}_ describe the main object in the images below?’, annotators choose between four responses: (i) yes, _{}_ seems to describe the main object, (ii) somewhat, _{}_ describes a similar object, (iii) no, but _{}_ describes another object common in the images, (iv) no, _{}_ is unrelated to the images.
The results of our human study are presented in Table 5, structured as follows: The first column is the rate that humans
picked any option aside from the fourth option. The second column is the rate that humans picked the first or second option.
The third column is the rate that the humans picked the first option. Also, we provide sample decodings for 45 randomly


|Model|Describes Concept Relevant Describes Concept Similar Describes Main Object to Images from Class to Main Object|
|---|---|
|Swin (S)
ResNet-50
Dino ViTs8|94.48%
84.60%
69.51%
95.14%
88.37%
71.36%
92.18%
76.45%
60.47%|
|**Average**|**93.93%**
**83.14%**
**67.11%**|
_Table 5._ Complete results from human study assessing quality of using concept-to-text to label classification head vectors from various
ImageNet trained models. We list the rate for which the decoded text satisfies some relation (e.g. ‘describes main object’) to the collage of
sample images shown for a given class.
|Model|Related vs. Similar vs. Dissimilar Main Object vs. Similar Unrelated vs. Unrelated vs. Dissimilar vs. Unrelated|
|---|---|
|Swin (S)
ResNet-50
Dino ViTs8|89.77%
76.63%
56.07%
91.27%
82.25%
56.87%
85.77%
67.13%
48.50%|
|Average
Random|88.94%
75.34%
53.81%
25%
11.1%
6.25%|
_Table 6._ Inter-annotator agreement, at various grains. _Random_ lists the agreement expected between two random annotators, computed as
1 _/n_ where _n_ refers to the number of possible choices for a given metric. We use two annotators for every query.
selected ImageNet classes in table 7. As shown in the main text, decoded concepts are almost always relevant to images
from the class. Interestingly, in about 10% of cases on average (obtained via subtracting column 1 from column 2), the
decoded word describes a common object in the images that is distinct from the main object. These cases correspond to
when the decoded word is a spurious feature to the class (e.g. ‘freeway’ for the class ’Water Tower’). Identifying spurious
features is a potential application of concept-to-text.
**Additional logistical details of study:** For each model-class pair, we obtain two responses. MTurk workers are compensated
$0 _._ 05 (USD) per task, resulting to an average rate of $15 per hour. We compute provide inter-annotator agreement for our
human study in table 6. Inter annotator agreement is generally high.


We note that the concept vectors we find are not always perfect. However, more refined concept vectors can be found by (1)
better prompt engineering and (2) extracting more relevant image samples to that concept.
The second point can be achieved by inspecting the most and least similar images retrieved for a desired concept, and
removing erroneous examples. Then, one can obtain a concept vector in the ordinary manner (i.e. training a binary classifier
in representation space to separate positive and negative examples of the concept), or more simply by taking the average of
their encoded representations.
Finally, we note that our method generally becomes better when any of the components which are involved in concept-to-text
and text-to-concept procedures become better. Indeed, better CLIP models, more powerful vision encoders, and improved
generative language models can contribute to an improved performance of concept-to-text and text-to-concept.


Some recent works also investigate bridging image and text models (Merullo et al., 2022; Zhai et al., 2021; Tsimpoukelli
et al., 2021), though their training mechanisms typically involve text supervision and propagating through both image and
text backbones, which can make them much more intensive. Previous efforts have sought to map image spaces to semantic
using cycle consistency objectives for zero-shot learning (Felix et al., 2018) or image translation (Zhu et al., 2017).
Our method of aligner training using representations from a pretrained model draws some parallels to knowledge distillation


(Hinton et al., 2015), where activations from a more powerful model are utilized in training a smaller one to behave similarly
(Beyer et al., 2021). However, the crucial difference between our work and standard knowledge distillation in that in our
method, the vision encoders we align to CLIP remain fixed. We do not wish to distill the knowledge of CLIP to other models
- in fact, we intentionally fix the off-the-shelf model and only allow for a minimal transformation (affine) of its representation
space. Instead, we argue that existing vision models already encode many human concepts in their feature space, though
accessing this information is challenging without a text encoder that maps to the same space. Our method allows for
interpretation of the off-the-shelf model’s space in an efficient and flexible way; i.e. by obtaining concept activation vectors
(CAVs) directly from text using CLIP’s text encoder. In summary, since the aim of our work is interpretability, we do not
wish to transfer or distill knowledge from one model to another. Rather, we seek to allow existing models to work with one
another in an efficient manner. We hope our work inspires others to investigate ways specialized models can be interfaced
together to accomplish novel ends in inexpensive ways.


|Class|Model Swin (S) ResNet-50 Dino ViTs8|
|---|---|
|American Alligator|Lizard
lizard
Florida|
|Messenger Bag|patch
pocket
tet|
|Spindle|spinning
spinning
coral|
|Radio|radio
radio
telephone|
|Ipod|screenshot
Nokia
USB|
|Yorkshire Terrier|photo
puppy
red|
|Hourglass|clock
clock
clock|
|Lion|lion
Lion
lion|
|Revolver|handgun
handgun
Glock|
|Scoreboard|sign
billboard
game|
|Wallaby|deer
bunny
shrew|
|Tent|Tibetan
cave
tent|
|Monastery|monastery
monastery
monastery|
|Front Curtain|small
Pluto
world|
|Golf Ball|a
golf
golf|
|Notebook Computer|laptop
Chromebook
notebook|
|Water Tower|large
tower
freeway|
|Gas Pump|vending
garage
garage|
|Smooth Newt|Lizard
lizard
slime|
|Platypus|mole
mole
crocod|
|Paintbrush|painting
paint
pen|
|Product Packet / Packaging|packet
is
Saturn|
|Chiffonier|replica
box
closet|
|Water Jug|jug
jug
cup|
|Boa Constrictor|python
python
python|
|Rapeseed|ﬁeld
yellow
farm|
|Police Van|police
van
police|
|Maltese|replica
puppy
Pluto|
|Pot Pie|previously
clam
pan|
|Menu|menu
menu
strawberry|
|Red Wine|red
red
red|
|Mosquito Net|young
bedroom
bedroom|
|Poncho|swarm
condom
square|
|Basenji|building
dog
puppies|
|Turnstile|human
gate
hospital|
|Sea Slug|coral
crocod
squid|
|Computer Keyboard|keyboard
keyboard
computer|
|Ballpoint Pen|3
glucose
neuron|
|Plate Rack|wall
table
table|
|Bridegroom|wedding
wedding
ﬂower|
|Fire Salamander|pair
Lizard
frog|
|T-Shirt|member
shirt
comet|
|Eastern Diamondback Rattlesnake|python
snake
python|
|Fiddler Crab|crabs
crabs
crab|
_Table 7._ Decodings (obtained via concept-to-text) of classification head vectors for 45 randomly selected ImageNet classes from three
pretrained models. The majority of decoded words are similar (though often broader) to the corresponding class object.


**Ziheng Li** **[1,]** _[∗]_ **, Shaohan Huang** **[2]** **, Zihan Zhang** **[2]** **, Zhi-Hong Deng** **[1,]** _[†]_ **, Qiang Lou** **[2]** **,**
**Haizhen Huang** **[2]** **, Jian Jiao** **[2]** **, Furu Wei** **[2]** **, Weiwei Deng** **[2]** **, Qi Zhang** **[2]**
1School of Intelligence Science and Technology, Peking University, Beijing, China
2Microsoft Corporation
{liziheng,zhdeng}@pku.edu.cn
{shaohanh, zihzha, qilou, hhuang, jiajia, fuwei, dedeng, qizhang}@microsoft.com

Dual-Alignment Pre-training for Cross-lingual Sentence Embedding

Ziheng Li
Shaohan Huang
Zihan Zhang
Zhi-Hong Deng
Qiang Lou
Haizhen Huang
Jian Jiao
Furu Wei
Weiwei Deng
Qi Zhang


Recent studies have shown that dual encoder
models trained with the sentence-level translation ranking task are effective methods for
cross-lingual sentence embedding. However,
our research indicates that token-level alignment is also crucial in multilingual scenarios, which has not been fully explored previously. Based on our findings, we propose a
dual-alignment pre-training (DAP) framework
for cross-lingual sentence embedding that
incorporates both sentence-level and tokenlevel alignment. To achieve this, we introduce a novel representation translation learning (RTL) task, where the model learns to
use one-side contextualized token representation to reconstruct its translation counterpart.
This reconstruction objective encourages the
model to embed translation information into
the token representation. Compared to other
token-level alignment methods such as translation language modeling, RTL is more suitable for dual encoder architectures and is computationally efficient. Extensive experiments
on three sentence-level cross-lingual benchmarks demonstrate that our approach can significantly improve sentence embedding. Our
[code is available at https://github.com/](https://github.com/ChillingDream/DAP)
[ChillingDream/DAP.](https://github.com/ChillingDream/DAP)


Cross-lingual sentence embedding encodes multilingual texts into a single unified vector space
for a variety of Natural Language Processing
(NLP) tasks, including cross-lingual sentence retrieval (Artetxe and Schwenk, 2019b) and crosslingual natural language inference (Conneau et al.,
2018). The text sequences can be efficiently retrieved and compared using the inner product between their dense representation.
The task of sentence embedding now heavily
depends on pre-trained language models (Devlin
_∗_ Work done during internship at Microsoft.
_†_ Corresponding Author.
(a) Sentence Alignment. (b) Dual Alignment.
Figure 1: Visualization of token representations of 100
Tatoeba sentence pairs from Arabic and English. The
high-dimensional vectors are projected onto a 2D space
by Principle Component Analysis. We show the results
of two models fine-tuned from multilingual BERT. The
model shown in Figure 1(a) only fine-tunes with the
translation ranking task, resulting in large misaligned
areas. This misalignment can be effectively eliminated
by the proposed RTL methods as shown in 1(b).
et al., 2019; Conneau and Lample, 2019; Conneau
et al., 2020b,a). By fine-tuning the CLS token
of the pre-trained model, they encode the input
text sequence into a single vector representation.
Recent research has shown that using the translation ranking task in combination with a dual pretrained encoder can result in superior sentence embeddings (Yang et al., 2019; Chidambaram et al.,
2019; Yang et al., 2021; Chi et al., 2021; Feng
et al., 2022). The purpose of fine-tuning the CLS
token is to learn sentence-level alignment and to
compress the entire sentence’s information into
the CLS token. This method makes the CLS tokens of semantically relevant sentences have larger
inner products. However, token-level alignment
in multilingual scenarios is also crucial, and the
fine-grained alignment task in cross-lingual sentence embedding has not been fully explored. As
shown in Figure 1, we visualize the token representation similarities between a pair of parallel corpora.
Training for an objective solely with regard to CLS
token causes the token representations to disperse
across the embedding space.
Based on our observations, we propose an efficient dual-alignment pre-training (DAP) framework for cross-lingual sentence embedding. The
embedding model is trained towards both sentencelevel alignment and token-level alignment. Previous cross-lingual pre-training studies (Chi et al.,
2021; Feng et al., 2022) employ translation language modeling (TLM) to achieve token alignment.
In this paper, we introduce a novel representation
translation learning (RTL) method that reconstructs
the entire English input based on the token representations of parallel non-English sentences using
a transformer model. By optimizing the RTL objective, the model learns to embed the information
of English sentences into the representation of its
non-English counterpart. Unlike TLM, computing
RTL only needs one-side self-contextualized representation and does not involve extra feedforward
propagation. We train our model on public corpora
and evaluate it on three cross-lingual tasks: bitext
retrieval, bitext mining, and cross-lingual natural
language inference. Our results demonstrate DAP
can effectively improve cross-lingual sentence embedding.
Our contributions are summarized as follows:
  - We propose a novel cross-lingual pre-training
framework DAP for sentence-level tasks,
achieving both sentence-level and token-level
alignment by representation translation learning, which is more suitable for dual encoders
and computationally efficient compared with
previous alignment methods.
  - Extensive experiments on three cross-lingual
tasks demonstrate DAP significantly improves
sentence embedding.
  - We train a model on a moderate-size dataset
and find its performance comparable with that
of the large-scale state-of-the-art pre-trained
model.


Following the success of BERT for English (Devlin et al., 2019), multilingual BERT comes out
by building a shared multilingual vocabulary and
training on multiple monolingual corpora with
the masked language modeling (MLM) objective.
XLM (Conneau and Lample, 2019) proposes a
translation language modeling (TLM) task which is
the extension of MLM to bitext corpora, so that the
model can learn the cross-lingual alignment from
translation pairs. Unicoder (Huang et al., 2019)
introduces three bitext pre-training tasks to help
the model capture cross-lingual information from
more perspectives. XLM-R (Conneau et al., 2020a)
scales up the amount of monolingual data and training time. They achieve better performance than
previous works without using parallel corpora.


The dual encoder architecture is first proposed
by Guo et al. (2018). They encode the source
and target sentences to a unified embedding space,
respectively, and compute the similarity score using inner product. The model is trained under a
translation ranking task to make the model score
higher for translation pairs than the negative examples. Yang et al. (2019) enhances the dual encoder by additive margin softmax, which further
enlarges the distance between negative pairs. Based
on additive margin softmax, LaBSE (Feng et al.,
2022) combines the translation ranking task with
MLM task and TLM task and trains on a larger
corpus. InfoXLM (Chi et al., 2021) interprets the
MLM, TLM and translation ranking task used in
cross-lingual pre-training in a unified informationtheoretic framework, based on which they propose cross-lingual contrastive learning to maximize
sentence-level mutual information.


**Transformer Encoder** Transformer encoder has
been widely adopted in modern language models (Vaswani et al., 2017; Devlin et al., 2019; Conneau and Lample, 2019). It consists of an embedding layer and _L_ stacked transformer blocks with
self-attention modules. Each input token _xi_ will
be encoded into a vector space as the initial hidden
vector _h_ [0] _i_ [. Then, in each transformer block, the]
hidden vector of the _i_ -th token _h_ _[l]_ _i_ [is computed from]
the self-attentive fusion of all hidden vectors output
from the previous layer:
_h_ _[l]_ = ( _h_ _[l]_ 1 _[, h][l]_ 2 _[,][ · · ·][, h][l]_ _S_ [) =] _[ f]_ _[l]_ [(] _[h][l][−]_ [1][)] _[.]_ (1)
We finally get the contextualized token representation _f_ ( _x_ ) = _f_ _[L]_ ( _f_ _[L][−]_ [1] ( _· · · f_ [1] ( _h_ [0] ))).
**Cross-lingual Pre-training** Masked language
modeling (MLM) (Devlin et al., 2019) and Transla
RTL head
(Pretrain Only)
Encoder
Figure 2: Workflow of the dual-alignment pre-training framework. We encode the bitext pair in dual encoder manner with a shared 12-layer transformer encoder and compute translation ranking loss and representation translation
loss using sentence representation and token representations respectively.
tion language modeling (TLM) (Conneau and Lample, 2019) are two typical tasks for cross-lingual
pre-training. MLM is conducted on monolingual
corpora. A randomly selected subset of input tokens will be replaced by a special [MASK] token or
another random token, and models learn to recover
these corrupted tokens according to the context.
TLM extends MLM to cross-lingual scenarios with
the following objective:
_LTLM_ ( _x, y_ ) = _ℓ_ ( _x ⊕_ _y, f_ ( _m_ ( _x_ ) _⊕_ _m_ ( _y_ ))) _,_
(2)
where _⊕_ denotes sequence concatenation operator
and _m_ denotes element-wise random replacement.
During training, models can predict the masked
token using the unmasked token in the translation.
In this way, models learn cross-lingual token-level
alignment using the parallel corpora.
However, TLM is designed for a cross-encoder
architecture in which tokens from the source and
target sentences are mutually accessible in intermediate layers. As a result, models trained with
TLM may rely on this information exchange, which
is not available during the inference stage when
sentences are independently encoded. Additionally, computing TLM requires an extra feedforward
propagation, which inputs concatenated sentence
pairs, resulting in increased training costs. Our proposed representation translation learning task can
overcome both the weaknesses.


Our dual-alignment pre-training framework contains two transformer models: dual encoder model
_f_ and representation translation learning (RTL)
head _g_ .
For the encoder model, we adopt the most popular BERT architecture with 12 layers of transformer encoder blocks, 12 attention heads, and
768-dimension hidden states. Following Devlin
et al. (2019), we prepend a special token [CLS] to
the input:
_f_ ( _x_ ) = _f_ ([CLS] _, x_ 1 _, . . ., xS_ ) _._ (3)
We take the hidden vector of CLS token _h_ _[L]_ _cls_
as the representation of the whole sentence _fs_ ( _x_ ).
Like other multilingual language models, our
model is language-agnostic, which means all languages share the same single transformer.
The RTL head is a stack of _K_ transformer encoder blocks with a vocabulary prediction head at
the top. The function of RTL head is to reconstruct
the translation sentence _y_ from the token representations of the source sentence _h_ _[L]_ (source sentences
indicate non-English sentences in this paper):
_g_ ( _h, y_ ) = _π_ - _W_ _[T]_ _g_ _[K]_ [ �] _g_ _[K][−]_ [1][ �] _· · · g_ [0] ( _h, y_ )��� _,_
_g_ [0] ( _h, y_ )=( _h_ _[L]_ 1 _[,][· · ·][,h]_ _S_ _[L]_ _x_ _[,]_ [[MASK]] _[,][· · ·][,]_ [[MASK]] ) _,_
~~�~~            - ~~�~~ ~~�~~
_×Sy_
(4)
where _π_ is softmax function and _W_ is the weight
matrix of the vocabulary prediction head. In our
experiments, we find a small RTL head with _K_ = 2
performs best generally.


To achieve both sentence-level and token-level
alignment, we design a pre-training framework
consisting of two tasks: translation ranking task
and representation translation learning task. These
two objectives are leveraged simultaneously during training. The whole procedure is depicted in
Figure 2.


Dual encoder models trained with the translation
ranking (TR) task have been proven effective in
learning cross-lingual embeddings (Yang et al.,
2019; Feng et al., 2022; Chi et al., 2021). These
models learn to maximize the similarity of the embedding pairs of parallel sentences and the dissimilarity of mismatched pairs. Therefore, they are
well suited for solving retrieval and mining tasks
that use inner product as ranking metrics. Following (Feng et al., 2022), we formulate the training
task as follows:
target, the reconstruction direction is always from
non-English sentences to their English translations.
Combining with the translation ranking objective
we get the final loss:
_LDAP_ = _LTR_ + _LRTL._ (7)
As RTL does not need an extra feedforward propagation, RTL only introduces a little computation
and will not slow down the pre-training significantly. The only time-consuming operation is the
softmax over the huge vocabulary which can be
further relieved by techniques like negative sampling and hierarchical softmax (not used in our
experiments).


In this section, we first describe the training setup.
Then we compare our method with previous works
on three sentence-level cross-lingual tasks.


Following Artetxe and Schwenk (2019b) we collect parallel training data for 36 languages (used in
XTREME Tatoeba benchmark) by combining Europarl, United Nations Parallel Corpus, OpenSubtitles, Tanzil, CCMatrix and WikiMatrix corpora,
which are downloaded from OPUS website (Tiedemann, 2012). As stated in section 3.3, we align
all other languages with English, so we only collect parallel corpora that contain English. For each
non-English language, we retain at most 1 million
sentence pairs at random. The whole dataset has
5.7GB data, which is far less than typical largescale pre-training (Feng et al., 2022; Chi et al.,
2021), but our method still achieves performance
comparable with the state-of-the-art.


We initialize the encoder model from multilingual
BERT base or XLM-R base, respectively, using the
checkpoint published on Huggingface model hub,
and initialize the _K_ -layer RTL head from the last
_K_ transformer layers by the corresponding encoder
model. The maximum sentence length is restricted
to 32 tokens, and sentences longer than 32 tokens
will be truncated. We train the model for 100,000
steps using the AdamW optimizer with a learning
rate of 5e-5 and a total batch size of 1024 on 8
Tesla V100 GPUs for 1 day. The results reported
are the average of three different seeds.
_LTR_ = _−_ [1]
_N_
_N_
- _e_ _[φ]_ [(] _[x][i][,y][i]_ [)]
log (5)
~~�~~ _B_ _[,]_
_i_ =1 _j_ =1 _[e][φ]_ [(] _[x][i][,y][j]_ [)]
where _B_ is the batch size and _φ_ ( _x, y_ ) is defined
as the similarity of the representation of each text,
typically _fs_ ( _x_ ) _[T]_ _fs_ ( _y_ ). In this paper, we use the
hidden vector of CLS token to represent the sentence.