Ranking-aware adapter for text-driven image ordering with CLIP

Thanks for your encouragement and insightful suggestions. We have included additional experiments and organized responses to address individual questions below.

[Q1] Different backbone and image resolution choices

Thanks for pointing this out and we conducted more experiments to explore the effect of different backbones for fair comparisons.

For object count sorting, we have already compared our method to the CLIP-based baselines, using identical settings (Table 2 of the main paper). In addition, we included InstructBLIP as a benchmark to evaluate the performance of a large vision-language model trained on extensive datasets with an extra-large backbone. We also explored the impact of different visual backbones and resolutions on our method. As reported in the table below, our method consistently achieves favorable performance compared to the CLIP baseline and InstructBLIP.

For the image quality assessment, facial age estimation, and HCI tasks, we conducted experiments with different vision backbones and image resolutions to compare our method with other approaches in the following table.

As shown in the table, using different backbones (with the same image resolution) has a slight impact on performance, e.g., ViT-B/16 even achieves slightly better results than ConvNext-L on KonIQA-10k and Adience. For the HCI dataset, we find that it is more sensitive to the backbone choice and image resolution, in which it can be attributed by its small training dataset size (1250 images) that would require more careful model tunings.

Nevertheless, our method still consistently outperforms the CLIP baselines and achieves competitive results against other methods that require a task-specific pre-training (e.g., VILA-R) or module (e.g., L2RCLIP). Specifically, L2RCLIP designs an additional module to learn ordinal representations from captions containing numerical information, such as "a photo of [1, 2, 3, ...] years old face.". While it can be effective for specific tasks, it cannot generalize to arbitrary scores, as required in tasks like image quality assessment (as discussed in Ln 53-84, 142-145).

Thanks for your insightful comments and suggestions.

[Q1] Technical contributions

Our proposed method focuses on designing a general framework for image ranking tasks guided by queried text prompts. We highlight our key contributions below:

1. We transform the image-text contrastive objective into a learning-to-rank objective in CLIP-based models, thereby allowing the model to directly learn the differences across image instances, rather than aligning the image one-by-one with its numerical value as in the prior work (see Figure 1).
2. We adapt the pretrained-CLIP with a lightweight ranking-aware adapter, which is a general design for various ranking tasks.
3. We propose to use two branches in our ranking-aware adapter: a regression branch for score prediction and an auxiliary ranking branch for learning image differences, which involves a relational ranking-aware attention module to capture text-conditioned visual differences across image pairs. Our ablation study (Table 5 of the main paper) demonstrates the effectiveness of these design choices.

Overall, our proposed method does not rely on task-specific modules for ranking tasks: 1) unlike existing CLIP-based methods (Paiss et al., 2023; Wang et al., 2023d; Ln 81-83, 142-145), our approach is not required to use prompts for specific numbers (e.g., "one cat", "two cats", etc.) and perform image-number matching, and (2) unlike the VILA method (Ke et al., 2023; Ln 451-454) designed for IQA, our method does not need additional task-specific pretraining or post-processing.

[Q2] Formatting issues

We have carefully addressed the formatting issues in the revised paper, including texts, equations, figures/tables, and citations.

• For the $V_{i}$ and $V_{j}$ in Eq(2), we revised the definition in the updated paper in Ln 253. The text-conditioned visual embeddings $z^{\prime}\_{i}$and $z^{\prime}\_{j}$ are utilized as values for computing the attention outputs, denotes as $V_{i} = z^{\prime}\_{i} \in \mathbb{R}^{p \times d'}$ and $V_{j} = z^{\prime}\_{j} \in \mathbb{R}^{p \times d'}$.
• For Figure 5 and Figure 6, we added the counts to the image.
• We revised the citation format in Ln 160-161, 186-187, 309, 319, 321, 416.
• We aligned the format of symbol $q$ in Eq(1) and Figure 2/Figure 3.
• We revised the symbol $P$ to $p$ to remove the confusion in Figure 3 and related contents as in Ln 243, 244, 252, and 354.
• We fixed the table margin for Table 1 and Table 2.

[Q3] Numbers of MLP block in the ranking branch

Despite a minor effect, we determine the number of MLP blocks empirically based on the performance. We adopt the same setting across all tasks and experiments.

[Q4]: Weights of the relational ranking-aware attention

The relational ranking-aware attention module is trainable. We have updated Figure 2 to make it more clear.

[Q5]: Handle multiple images

Our relational ranking-aware attention module does handle multiple images by capturing text-conditioned visual differences across image pairs. For example, if there are three images, it will calculate the visual embedding difference of 3 pairs ($I_1$ vs. $I_2$, $I_1$ vs. $I_3$, and $I_2$ vs. $I_3$). These embedding differences will then align to their ground truth relationships and can be optimized by using the pairwise ranking loss as in Eq(5), based on the output embedding $O_{ij}$ of our attention module (Figure 3).

[Q1] Core difference compared to the prior work (OrdinalCLIP, L2RCLIP, and NumCLIP)

As described in "Sec. 2 Learning-to-rank (Ln 142-145)" and Figure 1, unlike these existing CLIP-based methods that align the image one-by-one with its numerical value (e.g., "2 cats", "3 cats", etc.), our method directly learns visual differences across image instances with a general text prompt, e.g., "Sort images by the number of cats.".

This is achieved by our proposed learning-to-rank framework with the ranking-aware adapter, which is composed of two branches that are adopted for various ranking tasks: a regression branch for score prediction and an auxiliary ranking branch for learning image differences. Our ablation study (the last three rows in Table5) validates the effectiveness of our design choices.

[Q2] Comparisons with general VLM methods

For the object count sorting task, we have compared our method to the VLM models in Table 2 (also in Ln 425-426), e.g., VILA-VLM (Lin et al., 2023).

We also add another benchmark using InstructBLIP (Dai et al., 2023) for the object count sorting, image quality assessment (IQA), facial age estimation, and historical colored image dating (HCI) in the following table.

For each task, the prompt for the InstructBLIP is:

• Object count sorting: How many {category} in the image? Answer in a number.
• Image quality assessment: How is the photography quality of the image? Answer in a number from 1 to 10.
• Facial age estimation: How old is the person in the image? Answer in a number.
• Historical colored image dating: When was this image taken (from 1930s to 1970s)? Answer in a number.

For facial age estimation and HCI datasets, where ground truth labels are ordinal labels, we post-process the generated text into bins as defined by Wang et al. (2023d). As shown in the table, while InstructBLIP performs well in facial age estimation, it struggles with tasks such as image quality assessment and determining a photo's decade. In contrast, our proposed method performs better than the CLIP baseline under the same setting and the general VLM pretrained on large datasets.

Thanks for your valuable feedback and suggestions on the clarity of the paper. We respond to individual questions below.

[Q1] Symbol in Eq(5)

In Eq(5), $\Delta{O} = O_{i} - O_{j}$ which is $O_{i,j}$ defined in Eq(3). While the symbol $\Delta{O}$ is identical to $O_{ij}$, we remove $\Delta{O}$ to keep it consistent, and the paper is revised accordingly in Ln 273-275 of the updated paper.

[Q2] a) Role and application of the ranking score; b) Which of the two heads produces the final output? c) What exactly is being ranked in each task?

As mentioned in "Sec. 3.2 - Training objective and loss functions" (starting from Ln 264), the final output, i.e., the ranking score, is produced by the regression head for each image and thus the regressed scores can be used for ordering images in each task. On the other hand, the role of our proposed auxiliary ranking branch with relational ranking-aware attention is to enhance the representation by focusing on text-guided pairwise visual differences across images. As shown in our ablation study (the last three rows in Table 5), adding this auxiliary ranking branch improves the performance.

For the target to be ranked in each task, we have described them in "Sec. 4.1" and also listed below:

• Facial age estimation: the task is to rank images by the person's age group (8 groups). The prompt example is Sort images by the person’s age group..
• Historical colored image dating (HCI): the task is to rank images by the photo's taken date. The date category spans from 1930s to 1970s, corresponding to 1 to 5 in the experiment. The prompt example is Sort images by the taken date of the photo..
• Image quality/aesthetics assessment: the task is to rank the image by the photography quality, subjective preference, and objective properties like contrast and colorfulness. The values are normalized to 0.0 to 10.0. The prompt example is Sort images by the image quality of colorfulness..
• Object count sorting: the task is to rank images by the count of the queried object category. The value spans from 0 to more than a hundred. One prompt example is Sort images by the number of cat..

[Q3] Computation of mean absolute error (MAE)

While the regression head produces the ranking score, we follow the setting as in previous studies (Wang et al., 2023d) to compute the distance between ground truths and predicted scores for fair comparisons with the prior work. Specifically, the facial age in the Adience dataset is partitioned into 8 ordinal age groups with an index spanning from 1 to 8; the historical color image dating dataset has images spanning from 1930s to 1970s with an index spanning from 1 to 5. For computing the MAE, we simply compute the distance between the predicted ranking score and the index of the image related to the text query.

[Q4] Ambiguity in Terminology and Methodology

We use the term "reframe" to describe the transformation of CLIP’s paired image-text contrastive objective into a ranking task with a learning-to-rank objective. This transformation allows the model to directly learn visual differences across image instances, rather than matching a numerical value to an image as in existing CLIP-based methods (see Figure 1). As the "reframe" can potentially lead to confusion, we will rephrase the term to improve the clarity as the reviewer suggests.

Our paper adapts a pretrained CLIP model to rank images based on textual input, such as sorting images by properties like image conditions or quality. The proposed approach consists of three key components that differ from existing methods:

1. Transform the image-text contrastive objective into a learning-to-rank objective.
2. Introduce a lightweight ranking-aware adapter that fuses text embeddings into image embeddings and involves two branches: a regression branch for score prediction and an auxiliary ranking branch for learning image differences.
3. Develop a relational ranking-aware attention mechanism to capture text-conditioned visual differences across image pairs guided by their ranks.

Unlike existing methods that (1) align the image with its number, such as CLIP-based approaches that use image-number matching (Paiss et al., 2023; Wang et al., 2023d; Ln 81-83, 142-145), or (2) require specialized architectures with task-specific pretraining, like VILA (Ke et al., 2023; Ln 451-454), our method is a general framework and does not require additional task-specific pretraining or post-processing, and is optimized directly through learning visual differences across image pairs. We validate our method on diverse ranking tasks, showing favorable performance compared to CLIP-baselines and competitive results against state-of-the-art methods designed for each individual task.

Dear Reviewer,

Thank you for the comments on our paper. We have submitted the response to your comments. Please let us know if all issues have been addressed.

Best,
Authors