On the rankability of visual embeddings

We appreciate the reviewer’s detailed comments and insightful questions. We address the remaining concerns below.

W1: How were the nonlinear and finetuned methods trained?

We provide the training details in §3.2.3 below.

Nonlinear regression. We use a 2-layer MLP with 128 hidden dimensions and ReLU non-linearity (we will add this part). We randomly sample 30 hyperparameter configurations from a search space defined as follows. Learning rate and weight decay are sampled log-uniformly from ranges $10^{-6}, 10^{-1}$ and $10^{-7}, 10^{-4}$ respectively. Horizontal flipping augmentation is likewise switched on or off randomly. The learning rate is decayed to zero over the course of training using a cosine annealing schedule. We train for 100 epochs with a batch size of 128.

Finetuning. For ResNet50 and ConvNeXt, we conduct a grid search over 2 learning rates {10^{-4}, 10^{-3}}, and 2 weight decays {0, 10^{-5}}. For the other models, we conduct a wider grid search over 3 LRs {10^{-7}, 10^{-6}, 10^{-5}}, 4 weight decays {0, 10^{-5}, 10^{-4}, 10^{-3}}. The downstream learning rate is always fixed at 0.1, and we always use horizontal flip augmentations. We train all models for 20 epochs. Batch size was fixed at 128 except for ConvNeXt models and DINOv2 where it was reduced to 16.

Please let us know if further clarification is needed.

W2.1: Lacking references to existing literature on methods for constructing ordinal embeddings.

We discuss existing methods for constructing ordinal embeddings $57, 40, 20, 10$ in §2. In essence, these works are application-driven, aiming to modify CLIP’s embeddings or the encoder itself for specific ordinal regression tasks while we study whether general vision encoders already possess rankability with respect to arbitrary attributes. We thank the reviewer for mentioning $1, 2$ below. As with prior methods discussed above, $1,2$ train attribute-specific encoders (VGG16, ResNet18, and so on) for solving specific ordinal regression tasks. In contrast, our work aims to understand the linear, ordinal structure already present in the embedding space.

Other interesting future applications include:

• Zero-shot control in generative models: Rank axes from the encoder can transfer to latent control in generation tasks, offering an interpretable way to steer outputs along semantic attributes.

• Model auditing and diagnosis: When a model fails to predict certain continuous attributes, our method may provide an additional debugging tool for the intermediate representations to understand better why and how it fails.

We will add the discussion in §2.

$1$ Lee et al., Geometric order learning for rank estimation. NeurIPS, 2022.

$2$ Zha et al., Rank-N-Contrast: Learning continuous representations for regression. NeurIPS, 2023.

W2.2: Are the performance gaps between linear regression over trained embeddings still small when models are trained using the loss functions proposed in

$1$ , $2$ ?

This is a good point. While direct comparisons training the encoders used in our main paper using the loss functions from GOL and RnC were infeasible because of resource constraints, we evaluated regression over the models used in the original studies. Our interpretation of the reviewer's question is whether finetuning is a strong-enough upper bound for our analysis (please correct us if this is not the case); our goal is to evaluate this by comparing finetuning with MSE loss against the loss functions proposed in $1, 2$ .

Results on UTKFace using VGG16.

Results on AgeDB.

| Linear $ResNet18; DINO$ | Nonlinear $ResNet18; DINO$ | Finetuned $ResNet18$ | Rank-N-Contrast | | :---- | :---- | :---- | :---- | | 9.97; 8.18 | 9.50; 7.76 | 6.57 | 6.14 |

Observations and conclusions.

We observe that standard finetuning using MSE loss on downstream tasks (using the same or similar models,) yields performance quite close to that of $1, 2$ . On UTKFace and AgeDB, finetuned VGG16 and ResNet18 models achieve MAEs of 4.88 and 6.57, only modestly outperformed by $1$ (4.35) and RnC $2$ (6.14), respectively. For RnC $2$ , we use the ResNet18 model from timm while theirs was trained from scratch, however, for GOL $1$ , we used the same initial weights as $1$ .

We emphasize that our goal is complementary to GOL and RnC: while they propose methods for improving rank estimation, our study aims to evaluate rankability already present in off-the-shelf visual embeddings. Given our primary focus on probing intrinsic structure of pretrained embeddings using architecture-agnostic tools, and that finetuning itself already reduces the gap significantly, we conclude that finetuning provides a sufficiently strong upper bound for our analysis. We again thank the reviewer for mentioning these works, and will discuss them in our paper.

W3: does analysing ordinality of a generic encoder serve a real purpose given that dedicated ordinal encoders exist?

Yes, studying the presence of ordinality in off-the-shelf encoders has significant practical implications.

Current solutions for ordinal embeddings $57, 40, 20, 10$ require modification of the entire encoder or final embeddings, with dedicated loss functions or even architectural changes. This significantly limits adaptability. For each new attribute to rank a dataset, a user needs to employ a considerable amount of compute to train a dedicated embedding.

Our findings suggest that such a procedure is unnecessary. Many existing off-the-shelf encoders contain sufficient information to rank datasets according to common attributes. In fact, our approach to find a linear rank axis in frozen embeddings yields comparable performances against state-of-the-art embeddings designed to rank each attribute of interest $Table A12$ .

We will make this clearer in the revision.

W4: Lacking in-depth analysis on why certain attributes yield low rankability; unsupported hypotheses.

We hypothesize that rankability with respect to a given attribute depends on the variability of that attribute present in the training data (lines 189-193 in the submission). We present an empirical case study over the head-pose attributes using CLIP and its training dataset, LAION-400M.

Background. In our experiments using CLIP (obtained from timm, trained on LAION-400M), we find that rankability for pitch (SRCC = 0.91) is higher compared to rankabilities for yaw (0.31) and roll (0.40). We set out to verify whether the variability of the head orientation angles in LAION-400M face images follows the same trend.

Setup. We randomly sample 100 faces in LAION-400M using a face detection model $1A$ . We use the same model to estimate yaw, pitch and roll angles of each face. We quantify the variability of each attribute in LAION-400M dataset using the standard deviations of angles among the 100 faces.

Results and discussion. The standard deviations for pitch, roll and yaw are 21.2, 11.7 and 11.0 degrees, respectively, in LAION-400M. This concurs with the ranking of rankability among the three attributes: pitch > yaw > roll. This corroborates our hypothesis to a certain degree.

$1A$ Narayan, Kartik, et al. "Facexformer: A unified transformer for facial analysis." arXiv (2024).

W5: Number of datasets per attribute was limited to 2-3; Evaluation on more datasets?

Given practical limits in computational capacity, we strived to hit the right balance between the breadth of model choices (7 models), number of attributes (7 attributes), and number of datasets per attribute (2-3 datasets per attribute where available), which already surmounts to 150 GPU days of compute (§3.2.4).

At the same time, we agree that additional datasets will further support our claims. We present additional results using CLIP-ViT-B/32 on the datasets referenced by the reviewer. We will include the full results in the camera-ready version.

We confirm the finding in our work that rankability is observed for the suggested datasets too.

Below we also report the transfer results. For every age dataset we calculate two metrics:

• SRCC of rank axis obtained using the same dataset. We train on the dataset itself, then test on a held-out split from that same dataset. This is the first numeric column in the table.

• Mean transfer SRCC from other datasets. We train five separate rank-axis models, each on one of the other datasets; evaluate all five on the current dataset and take their average SRCC. This corresponds to the second column.

For brevity we show only this averaged “Mean transfer SRCC from other datasets” in the table; the full per-source breakdown will be included in the camera-ready version.

As in the main paper, we confirm non-trivial transfer of rank axes across age datasets.

Questions

Q1: Were upper-bound models trained using MAE or MSE loss?

We used the MSE loss across all settings.

Q2: How many trials were used to report the few-shot results?

We conducted three random trials. We plot the standard deviations in our results – the variation across trials gets smaller as we approach the “95% of full-training performance” point.

Q3: Experiments on additional datasets.

See W5 above.

We thank the reviewer for the careful reading and constructive questions. We hope that the additional experiments and clarifications below address the remaining concerns.

W1: The claim of linear embedding is unverified.

1. What about Pearson linear correlation coefficient (PLCC)?

We wish to clarify two related but distinct notions of linearity in the context of ordinal attribute embeddings:

• Linearity of the rank axis refers to whether there exists a consistent direction in the embedding space along which an ordinal attribute (e.g. age, brightness) increases. Our work focuses on identifying this direction. To evaluate this, we compute Spearman’s rank correlation coefficient (SRCC) between the true attribute ranks and the projections of embeddings onto the identified rank axis.

• Linearity of the ordering along the rank axis concerns whether items are spaced in a regular, linear fashion along this direction. For example, if the true attribute values increase by a fixed amount, do the corresponding embedding projections also show equal spacing? This reflects the regularity or uniformity of the embedding distribution along the rank axis.

While our primary focus is on (1), we agree that further analysis of (2) could reveal useful insights into how ordinal attributes are represented in embedding space.

We report PLCC for the CLIP-ViT-B/32 embedding against 5 attributes below. For most attributes, the PLCC measures are close to SRCC (0.921 vs 0.932 for head pitch). This indicates that the ordering of attributes along the rank axis is largely linear. This is an interesting further insight into the structure of the embeddings. We will include the PLCC versions of Tables 2 & 3 in the paper and discuss the implications. We thank the reviewer for the additional insight.

2. Algorithmic operations in the embedding space: does moving by the same magnitude and direction yield the same effect?

Yes, to some degree. The Pearson's correlation results above show this indirectly. We show this more directly with a dedicated experiment below.

Setup. We measure the calibration of the rank-axis projected distances of the embeddings against the ground-truth ordinal attribute. We use CLIP-ViT-B/32 embeddings on the ShanghaiTech-B and UTKFace datasets for the age and crowd count attributes, respectively. We then measure the linearity of the relationship between the ground-truth age and the projected distances using the coefficient of determination ($R^2$).

Results and conclusion. The $R^2$ values are 0.761 for crowd count and 0.849 for age. This indicates a largely linear agreement between projected distance along the rank axis and the ground-truth attribute values.

W2: Few-shot claims are over-stated.

We thank the reviewer for pointing this out. We will revise all mentions of “few-shot” in the paper to a more accurate wording like “sub-1k supervision” or “5-10% of the dataset” to avoid confusion and false expectations.

W3: Behavior of out-of-scope values: what happens if the inference image is outside the range of training images?

We thank the reviewer again for a very interesting question. This will shed further light on the structural regularity of the embedding space. We design an experiment below for the extrapolation behaviour.

Setup. The UTKFace dataset contains age labels between 21 and 60. We split the dataset into two subsets: one with ages 21-50 and the other with ages 51-60. We test whether the rank axis obtained over the 21-50 split applies to the 51-60 split by computing the SRCC against the ground-truth labels on the 51-60 split. As a control, we compare against another rank axis computed by the full age range 21-60.

Results and discussion. We observe that the rank axis obtained on ages 21-50 achieves an SRCC of 0.174 on the 51-60 set, compared to 0.267 when trained on the full range 21-60. Extrapolation is indeed harder than interpolation. It is thus recommended that users obtain extreme examples to cover the full dynamic range of attributes.

W4: Comparisons are missing.

1. What about SOTA methods?

We already have comparisons against the state-of-the-art models in Table A12 of the supplementary material. We observe that On UTKFace (age estimation), our rank axis over fixed embeddings achieves an MAE of 4.25, slightly underperforming [1] (MAE = 3.83) and performing on par with [2] (MAE = 4.23). This is remarkable given that [1, 2] rely on further modifications to the embeddings designed to enhance ordinality. On UCF-QNRF (crowd-counting), likewise, our approach achieves an MAE of 246.4, even outperforming CrowdCLIP [3] a model designed for crowd counting.The results further support our thesis that pretrained vision embeddings are highly rankable, even compared to more complex specific systems designed for the prediction of respective attributes. We will include this discussion in the main paper.

2. Comparisons with image quality assessment works like CLIP-IQA [55] that discuss similar ideas.

We thank the reviewer for mentioning CLIP-IQA. In Table A12 of the supplementary material, we include comparisons with CLIP-IQA. On KonIQ-10k (aesthetics), our linear rank axis achieves an SRCC of 0.860, outperforming CLIP-IQA (0.727). We emphasise that this comparison is not entirely direct: CLIP-IQA does not use supervised training data and relies solely on prompt-based scoring. However, we agree that such comparisons provide further contextualization of our work. We will include more of such quantitative comparisons in the main paper, instead of supplementary materials.

Limitations: the paper has not discussed limitations or potential negative social impact.

While we discuss the limitations in lines 189-193, we agree that we should discuss the potential negative societal impacts of our work. We recognize a few potential issues. First, the ability to rank images containing personal or sensitive information (e.g. faces) with respect to arbitrary attributes may risk exposing certain individuals that are at the extreme ends of a spectrum (e.g. income level). Second, ordering individuals along a single attribute axis, such as gender, may reinforce existing stereotypes and offend and marginalize certain demographic groups. This discussion will be added to the manuscript.

Dear Reviewer nvX2,

Thank you for your earlier note indicating that you would consider increasing your score following discussions on the concerns raised in your review.

We have provided a detailed response to address these points, but have not yet heard back from you. As we are now late in the discussion phase, we would appreciate knowing whether our clarifications have resolved your concerns and whether you are considering an increase in your score, or if there are any remaining issues we should address while time still allows.

Best regards,

Authors

We thank the reviewer for their recognition of our findings and for their commendation of our experimental setup. Below, we address the remaining concerns.

W1: Theoretical explanations: why do visual embeddings possess good linear rankability, and why do CLIP models typically perform better in this regard?

While theoretical analysis was outside of the scope of our study, we conjecture that rankability along a given ordinal axis is directly related to variability of different attributes in the training data (lines 189-193). The wider the attribute distribution, the stronger the optimization pressure may be for rankability to emerge. To confirm this, we conduct a case study using CLIP.

Background. In our experiments using CLIP (obtained from timm, trained on LAION-400M), we find that rankability for pitch (SRCC = 0.91) is higher compared to rankabilities for yaw (0.31) and roll (0.40). We set out to verify whether the variability of the head orientation angles in LAION-400M face images follows the same trend.

Setup. We randomly sample 100 faces in LAION-400M using a face detection model $1A$ . We use the same model to estimate yaw, pitch and roll angles of each face. We quantify the variability of each attribute in the LAION-400M dataset using the standard deviations of angles among the 100 faces.

Results and discussion. The standard deviations for pitch, roll and yaw are 21.2, 11.7 and 11.0 degrees, respectively, in LAION-400M. This concurs with the ranking of rankability among the three attributes: pitch > yaw > roll. This corroborates our hypothesis to a certain degree.

$1A$ Narayan, Kartik, et al. "Facexformer: A unified transformer for facial analysis." arXiv (2024).

As for why CLIP models perform better, we conjecture that label noise present in captions may play a role. Below, we reason via an example.

When training a CLIP model on image-caption pairs that include age-related descriptors, there may often exist ambiguity in the use of age terms. For instance, middle-aged individuals may sometimes be described as “young”, while young individuals are described as “middle-aged.” However, one rarely describes young individuals as “old.” This asymmetric labeling noise may implicitly encourage the model to embed images such that the representations of “young” and “middle-aged” individuals are closer to each other than either is to that of “old” individuals, i.e.,

$$\text{dist}(\text{old}, \text{young}) > \text{dist}(\text{middle-aged}, \text{young}).$$

This may, in turn, indirectly produce the ordinal structure observed in our study.

---

W2: Experimental results

A. Finetuned results for some datasets like ST-A and ST-B are even worse than their Nonlinear counterparts.

This is a valid concern. In principle, finetuning should outperform regression on a frozen encoder. However, finetuning is highly sensitive to hyperparameter choices.

That said, for smaller datasets such as ST-A and ST-B, the restricted hyperparameter range may lead to overfitting during finetuning. As a sanity check, we conduct a wider hyperparameter search (four times as long as the original search) for CLIP-RN50 on ST-A and ST-B, with results presented below (model selection using validation SRCC). We observe that with a wide enough parameter search, finetuning does outperform linear and nonlinear regression, albeit still not drastically (~0.06 SRCC in each case). For most datasets, finetuning slightly outperforms linear and nonlinear regression by default; while the current sanity check indicates that even longer hyperparameter searches would not have altered our conclusions drastically.

---

B. How do different dataset types, attribute types and training loss functions affect rankability?

We believe that rankability emerges out of the interaction between the attribute, the evaluation dataset, and the training objective.

Effect of attribute and dataset. While we observe non-trivial rankabilities for all attributes and datasets (Table 2 in the paper), some patterns emerge. Age, crowd count and pitch angle are ranked better than image aesthetics and recency, while pitch is ranked significantly better than yaw and roll (lines 183-187). We validate our hypothesis around this observation in W1 above. In addition, we also observe that within the same attribute (age), Adience is ranked better (0.723) than UTKFace (0.623). We believe that this stems from the fact that labels are coarser in Adience with 8 age group labels instead of exact year as in UTKFace. We also invite the reviewer’s feedback on

• our analysis on other attributes under Reviewer pn5U’s W1, and
• our evaluations on other datasets under Reviewer SNnm’s W5.

Effect of training objective. We consider 7 pretrained vision encoders. CLIP encoders were trained using the standard InfoNCE loss. Among the remaining encoders: DINOv2 was trained using self-distillation loss, ConvNeXtv2 using self-supervised FCMAE loss, and RN50 and ViTB32 were trained using supervised classification.

• We observe that CLIP is, in general, more rankable than or at par with non-CLIP encoders on all attributes except for yaw and roll where DINO performs best (lines 175-182). We observe a mixed, attribute-dependent trend among these four encoders.
• On age, both self-supervised models DINO and CNX perform on par with each other while outperforming the fully supervised models. On crowd count, pitch, aesthetics and image recency, CNX outperforms DINO to varying extents.
• The fully supervised models win out over the self-supervised models on crowd count, while underperforming on all other attributes (when the best models out of both groups are considered).

In conclusion, self-supervised training objectives seem to lead to better rankability than supervised ones, while CLIP's training outperforms both. Future studies may also consider the role of pre-training datasets.

---

W3: Method design: do efficient methods for finding rank axes generalize to more difficult datasets like Kinect?

To check this, we conducted experiments using Kinect (pitch) on CLIP. We report results in the table below. We observe again that using just 512 samples (~5% of the full dataset), we already recover an SRCC of 0.9, i.e., more than 95% of the SRCC obtained upon training on the full training set.

---

Questions

Q1: Theoretical analysis

See W1 above.

Q2: Why does Finetuned sometimes perform worse than Nonlinear?

See W2.A above.

Q3: Impact of different factors on rankability

See W2.B above.

Q4: Effectiveness of efficient rank-axis finding methods on more challenging datasets

See W3 above.

We sincerely thank the reviewer for their thoughtful and encouraging feedback. We are especially glad that the reviewer finds the framing of rankability as a property for understanding embedding spaces novel and well-motivated. We appreciate the suggestions for expanding attribute diversity and applications. We address each point below.

W1: Attribute diversity

1. Attributes are face-related or low-level; generalizability is questionable.

We appreciate this observation. Broadly, our study focuses on four categories of attributes:

• Age progression.
• Geometric understanding (head pose).
• Scene complexity and counting (crowd count).
• Image quality (aesthetics, modernness).

Our reliance on continuously annotated attributes naturally constrains us to this selection. However, as the reviewer notes in the following section, a qualitative exploration of other attributes is also interesting; we address this next.

2. Qualitative study on more difficult attributes such as color, size, and semantic attributes like "luxurious".

These experiments would indeed add meaningfully to the empirical breadth; we thank the reviewer for proposing them. Below, we present the results.

• Ordinal attributes on dSprites: color, scale, position, orientation

dSprites provides a synthetic testbed for rankability along more "difficult" attributes like color and size.

Setup. We use pretrained CLIP ViT-B/32 embeddings to analyze the rankability of 5 attributes: hue, x-position, y-position, scale, and orientation. For hue, we consider the segment between red and yellow to avoid cyclicity of the attribute. We compare the no-train, linear, non-linear, and finetuned performances.

Results and discussion. The table above shows the results. Hue is already well-represented in untrained representations (no-train, 0.919), whereas CLIP pretraining does not seem to boost ordinality (linear, 0.920). We conjecture that hue is often not a discriminatory signal in caption-based contrastive training. Other attributes are significantly better represented (e.g. 0.151 no-train $\\rightarrow$ 0.716 linear for orientation). For some, frozen embeddings are as rankable as finetuned ones; for example, scale ordering using rank axis (0.980) vs dedicated fine-tuning (0.986). CLIP embeddings are inherently capable of ranking these attributes.

• Qualitative study: “Luxuriousness”

We study the rankability of CLIP with respect to "luxuriousness" using images from the kitchen room & dining room table class in Open Images V7.

Setup. We randomly sample 20 pairs of images and ask three annotators to answer which image in the pair is more "luxurious". The "ground truth" luxuriousness is determined via majority vote. We measure quality of the rank axis by measuring accuracy of the prediction of the more luxurious image in each pair. The random chance baseline is 0.50 and the upper bound is given by average inter-annotator agreement. We repeat this process over 3 independent trials with different sets of 20 pairs.

Obtaining a rank axis from visual extremes. We prompt GPT-4o to generate two synthetic reference images: one depicting a minimal dining room table setup, while another represents a visually luxurious version of the same scene. We compute the rank axis as the L2-normalized difference between their CLIP image embeddings.

Results. The model achieves an accuracy of 0.75 ± 0.07, well above the random chance accuracy of 0.50 and at the same level as the average inter-annotator agreement of 0.73 ± 0.10. CLIP encodes linear ordinal structure to support ranking along semantic attributes like luxuriousness. As shown in the main paper, the rank axis can easily be obtained using just two extreme samples.

W2: Impact of attribute-wise variety in the training data; why does DINO outperform CLIP on yaw and roll; on which attributes would CLIP significantly outperform DINO?

We thank the reviewer for highlighting this direction. As a preliminary test for our hypothesis, we conducted a case study using headpose angles on CLIP.

• Case study on headpose angles with CLIP

We hypothesize that rankability w.r.t. a given attribute depends on its variability in training data (lines 189-193). We present an empirical study over head-pose attributes using CLIP and its training dataset, LAION-400M.

Background. In our experiments using CLIP, we find that rankability for pitch (SRCC = 0.91) is higher compared to yaw (0.31) and roll (0.40). We verify whether the variability of the head orientation angles in LAION-400M face images follows this trend.

Setup. We randomly sample 100 faces in LAION-400M using a face detection model $1A$ and the same model to estimate yaw, pitch and roll angles of each face. We quantify variability of each attribute using standard deviations among the 100 faces.

Results. The standard deviations for pitch, roll and yaw are 21.2, 11.7 and 11.0 degrees, respectively, in LAION-400M. This concurs with the ranking of rankabilities: pitch > roll > yaw. This corroborates our hypothesis to a certain degree.

$1A$ Narayan, Kartik, et al. "Facexformer: A unified transformer for facial analysis." arXiv (2024).

• CLIP vs DINO

DINO outperforms CLIP on yaw and roll. CLIP relies on language supervision. As long as the embedding space places similar image-text pairs together and dissimilar ones apart, there is limited incentive for fine-grained understanding of the image. We hypothesize that LAION-400M also has far fewer captions describing yaw and roll. Intuitively, for yaw, “looking to the left/right” can be ambiguous, and it is even more unnatural to describe roll using language. We verify this intuition empirically by counting the frequency of such phrases in LAION captions: out of ~13M captions, pitch is described in ~800k captions, yaw in ~5k captions while roll is only described in ~30 captions.

Hence, while there is some incentive to learn pitch, there are reduced incentives for learning yaw and roll.

DINO relies on self-distillation for training, i.e., without language guidance. Additionally, the training process involves several augmentations. As such, fine grained understanding of the visual structure in the image becomes important, likely leading to enhanced coverage of attributes like yaw and roll.

CLIP outperforms DINO on image aesthetics and recency. We observe that CLIP significantly outperforms DINO on image-level attributes like aesthetics and recency. We hypothesize that this can again be explained on the basis of attribute variability. CLIP was pretrained on a largely uncurated dataset (LAION) with very basic caption filters in place. This would have allowed varying qualities of images into CLIP’s training pipeline, leading to high variation in quality. On the other hand, DINO’s training dataset, LVD-142M, is much better curated, and likely contains only high quality images. Without variation in quality, DINO does not learn much about this attribute.

W3: Applicability

1. Is there any other application other than fast database sorting? Is rankability a good thing and should we train visual encoders to have it?

Higher rankability, by definition implies we may perform continuous regression without changing the underlying embedding. Current works on ordinal embeddings ( $57, 40, 20, 10$ among others), make dedicated efforts to refine the embedding space for ranking-awareness with respect to one attribute at a time. After applying the ranking-aware finetuning, it is often also unclear whether the same model can then continue to support other downstream tasks.

In contrast, our work studies whether existing vision encoders linearly encode general ranking-awareness (i.e., with respect to several attributes) even without dedicated optimisation pressures or post-training modifications. Indeed, we find that vanilla CLIP embeddings, for instance, are already ordinal with respect to many attributes studied in prior works.

Strikingly, simple linear regression over frozen embeddings often performs on par with these dedicated ordinal embeddings (we report detailed SOTA comparisons in Table A12 of the supplement).

We believe that this observation in itself is surprising and practically valuable, as it eases all the ordinal regression tasks that would otherwise be handled by dedicated ordinal encoders.

$3A$ Yu et al. "Ranking-aware adapter for text-driven image ordering with CLIP." ICLR 2025.

2. If model A outperforms model B on rankability over attribute X, will the trend transfer over to classifying on the basis of X?

We expect that the existence of a rank axis should also help classification. Below, we conduct a preliminary experiment. As expected, the better regression model is also the better classification model.

Q1: Expand attribute diversity. See W1 above.

Q2: Downstream applications; is rankability good? See W3 above.

Q3: Training vision encoders for rankability

1. Should we train for rankability?

See W3 above.

2. How can we train for rankability?

Our study finds that rankability naturally emerges in many vision encoders, especially modern ones (e.g., CLIP). However, given the benefits of rankability, future work may also explore ways to induce it explicitly, perhaps, by including pairwise comparison annotations (image 1 > image 2 for a given attribute), and adding a hinge loss to the pretraining objective.