Conditional Representation Learning for Customized Tasks

Thanks for your recognition of the problem this work aims to address, as well as our simple yet effective idea. We sincerely appreciate your positive feedback. Below are the point-by-point responses to your concerns mentioned in the weaknesses and questions sections.

Weakness 1: Best set of textual basis

We appreciate the insightful comment. We will try to adopt a unified selection strategy in this work (e.g., variance, distance, PCA, maximal connected subgraph, etc.). We hold the same view that this is a valuable direction, and we will consider it in future explorations.

Weakness 2: Quality of the generated texts

Our method only requires that the generated texts roughly describe the given criterion, without the need for precise or exhaustive phrasing.

Take the "color" criterion as an example, the generated texts are:

["red", "crimson", "scarlet", "ruby", "cherry", "rose", "burgundy", "cardinal", "wine", "firetruck red", "ferrari red", "carmine", "sangria", "apple red", "poppy", "raspberry", "red", "pink", "blush", "fuchsia", "magenta", "hot pink", "baby pink", "pink", "coral", "peach", "apricot", "salmon", "light pink", "powder pink", "lavender", "violet", "indigo", "purple", "amethyst", "grape", "orchid", "eggplant", "plum", "lilac", "mauve", "periwinkle", "blue", "navy", "azure", "sky blue", "cobalt", "cerulean", "sapphire", "electric blue", "royal blue", "baby blue", "powder blue", "blue", "teal", "turquoise", "aqua", "seafoam", "green", "emerald", "forest green", "mint", "lime", "olive", "kelly green", "sage", "pea green", "jade", "chartreuse", "green", "yellow", "gold", "lemon", "butter", "canary", "mustard", "sunflower", "cream", "ivory", "beige", "tan", "coffee", "chocolate", "yellow", "brown", "copper", "rust", "mahogany", "walnut", "espresso", "gray", "slate", "charcoal", "pewter", "silver", "platinum", "black", "coal", "ebony", "jet black", "onyx", "snow", "ivory", "white", "pearl", "bone", "eggshell", "vanilla"]

Some generated texts are duplicated ("red", "blue", etc.), and we'll filter out these redundant texts. As the table below shows, this brings a slight improvement in clustering performance.

As for the redundancy, we explore the number of generated words in Figure 6 in the main text. As depicted, our method achieves stable performance, demonstrating its robustness to the redundancy of the generated words.

Weakness 3: More appropriate title

Thank you for pointing this out — we agree that framing the classification evaluation as “few-shot learning” is more precise than “supervised linear probing”. We will modify it in the next version.

Question 1: Comparison to compositional retrieval

The customized similarity retrieval task is more closely associated with directed representation transformation, where the criterion is fixed.

In comparison, compositional retrieval does not involve such a targeted transformation or toward a specific criterion. Instead, it typically involves retrieving images from the gallery based on the combination of multiple texts and the given image.

In future work, we also plan to explore how our method can be extended or adapted to compositional retrieval scenarios.

We will add the above discussions in the next version.

Finally, we sincerely thank you for your positive feedback and insightful comments again, which are encouraging and helpful in guiding the future development of our work.

We are grateful for your constructive and kind comments. Below, we provide the point-by-point response to your concerns.

Weakness 1 & Question 2: Our contribution

We clarify that our proposed conditional representation learning is remarkably distinct from works on task-conditioned and goal-conditioned representations. In essence, our method operates at a more fundamental and general level compared to task- or goal-conditioned approaches..

We first make a summary of our work and task/goal-conditioned works as follows:

1) Task-conditioned works aim to learns representations that reveal the underlying correlations among different tasks. For example, taskonomy computes the optimal transfer learning paths among tasks (point matching, reshading, etc.) to minimize the amount of required annotation.

2) Goal-conditioned works aim to learns representations that meet the required outcomes or goal states. For example, Robotics/RL works typically involve using the goal state as a conditional input to learn policies to instruct the agent from its current state to the goal state.

3 ) Our work aims to learn representations that adapt to arbitrary user-specified criteria ("color", "texture", etc.), extending the conventional representation learning that focuses on a single dominant aspect (typically “category”).

From the above comparison, we highlight the key difference:

Task-conditioned works focus on generalization across tasks and are commonly used in supervised multi-task learning. Goal-conditioned works, on the other hand, emphasize generalization across goal supervision and are typically applied in reinforcement learning. In contrast, our work targets generalization across user-specified criteria, which is situated in the context of unsupervised representation learning, a more fundamental and upstream field.

We will clarify this positioning more clearly in the introduction section.

Weakness 2 & Question 3: Task realism (Framing existing tasks under this setting)

The proposed concept of conditional representation learning holds significant promise and research merit. Existing representation learning works primarily focus on learning a representation under a universal criterion ("shape" or "category"). However, this may not always align with customized downstream tasks. For instance, in animal habitat analysis, researchers prioritize scene-related features, whereas universal embeddings emphasize categorical semantics. Moreover, in the fashion retrieval scenario, users often search for items based on different criteria, such as color, material, or occasion. A user might look for "red dresses" today, "formal outfits" tomorrow, and "lace tops" the next day. While the input image remains the same, the relevant features for retrieval vary depending on the user's intent. Conditional representation learning enables generating representations that are tailored to the given criterion conveniently and quickly, improving user satisfaction.

We propose conditional representation learning and offer an efficient and generalizable solution. We utilize it as a plug-and-play module to apply to four downstream tasks: linear probe (Section 4.1.1), clustering (Section 4.1.2), customized similarity retrieval (Section 4.2.1), and fashion retrieval (Section 4.2.2). In fact, they are not self-made tasks. To be specific, these four tasks are derived from classification and retrieval, the most fundamental tasks to evaluate representation learning.

Weakness 3: Discussions on LLM-based descriptor generation works

The first paper you mentioned (paper 1) belongs to prompt learning, and we discuss these works in Question 6.

The second paper (paper 2) in the comment is similar to LaBo and LM4CV in our Related Work section (line 109), belonging to interpretable classification.

Compared to these two papers, CRL differs in the following two aspects:

1) Generated content: They use LLM to generate content to describe each class name, implicitly related to universal "category" criterion. For example, paper 1 asks the LLM to generate "two legs", "a small head", etc., for the class "hen". In comparison, our work uses LLM to generate content to describe the criterion, which is a higher-level concept. The generated results can be understood as class names, which can be found in our appendix.

2) Purpose: Paper 1 aims to improve classification accuracy and interpretability under the universal criterion. Paper 2 aims to learn the optimal prompt to improve zero-shot image classification accuracy. In contrast, our work focuses on obtaining conditional representations across different criteria.

Question 1: Comparison to text-conditioned representation methods

The baselines (Combiner, LinCIR, etc.) in Section 4.2.1 are exactly the methods you mentioned. These methods obtain image representations conditioned on text already, and then use the new representations to retrieve the most similar images from the gallery.

Compared to these methods, CRL does not require task-specific design and can be used in a plug-and-play manner. Nevertheless, CRL needs to be constructed on models where image and text representations are made separately, thus it’s hard to apply CRL to backbones that are not inherently designed for this type of architecture.

By the way, we apply CRL to BLIP2 and achieve consistently ideal performance improvement like CLIP, which further demonstrates the generalizability of CRL.

Question 4: With how many examples do the improvements level off

Firstly, we would like to clarify that CRL itself is unsupervised and does not rely on additional labeled examples. When applying to downstream tasks, the training data we use is the same as the corresponding baselines.

We understand that you might be referring to the number of texts generated by the LLM. We have explored this in Section 4.3. As Figure 6 indicates, when the number of generated texts increases to around 100, the performance improvements level off across all four tasks.

Question 5.1: Input for the evaluated tasks

The input for the evaluated tasks includes the criterion and the LLM/VLM prompt.

Question 5.2: Method Robustness

We now present an analysis of CRL's robustness for these factors.

Robustness against different LLMs

As for the selection of LLMs, we would like to clarify that our method does not particularly rely on any specific LLM. To be specific, we use the same prompt to query four mainstream LLMs, GPT-4o, Deepseek-v3, Gemini 2.5, and Claude 4.

The result demonstrates that CRL remains stable when adopting different LLMs to generate the texts.

Robustness against different LLM prompts

As for the LLM prompt, we devise 5 different templates:

1) Generate common expressions to describe the [criterion].

2) List a wide variety of typical phrases used to characterize the [criterion].

3) Enumerate familiar terms or expressions people often use when referring to the [criterion].

4) Identify and list expressions frequently used to convey the concept of the [criterion].

5) How do people usually talk about the [criterion]?

The table above suggests that CRL is robust against the LLM prompt.

Robustness against different VLM prompts

As for the VLM prompt, we also devise 5 different templates:

1) objects with the [criterion] of [text]

2) a photo with the [criterion] of [text]

3) itap with the [criterion] of [text]

4) art with the [criterion] of [text]

5) a cartoon with the [criterion] of [text]

As suggested in the above table, CRL achieves stable performance across different VLM prompts.

Question 6: Comparison to visual prompt learning methods

We summarize the differences between our work and prompt-learning methods as follows:

1) Tasks: prompt learning methods aim to learn an optimal prompt to solve the classification task. By comparison, our method serves as a plug-and-play module for the conditional representation learning task.

2) Priors: prompt learning methods rely on the prior of class name, while CRL does not require it. Obtaining such priors is impractical in domains featuring massive classes. For example, in the fashion retrieval task, the DeepFashion dataset consists of 1000 classes. In real-world applications, it's even harder to make class names as priors since they are dynamic.

Limitations Discussion

Despite CRL's generalizability across different criteria, it may not outperform clustering methods like CC and SCAN under the universal criterion "shape". This is likely because these methods employ specially targeted designs for clustering under this criterion.

We will add the above results and discussions in the next version.

Finally, we deeply appreciate your time and constructive feedback, and we earnestly hope that our response meets your expectations and resolves the issues raised.

Dear reviewer YAxP,

We sincerely appreciate the time and effort you have devoted to reviewing our submission. We are writing to kindly follow up on your review comments. We would truly appreciate any further feedback you could provide, as we are eager to incorporate your perspective and improve the quality of our work. Please let us know if there’s anything specific you would like us to address.

We appreciate your careful comments. However, we believe that the judgment may overlook key aspects of our contribution, which we would like to clarify below.

Weakness 1.1: Technical novelty

The concept of conditional representation learning we propose is of high potential and research value. Existing representation learning works primarily focus on learning a representation under a universal criterion ("shape" or "category"). However, this may not always align with customized downstream tasks. For instance, in animal habitat analysis, researchers prioritize scene-related features, whereas universal embeddings emphasize categorical semantics. Moreover, in the fashion retrieval scenario (Section 4.2.2), users often search for items based on different criteria, such as color, material, or occasion. A user might look for "red dresses" today, "formal outfits" tomorrow, and "lace tops" the next day. While the input image remains the same, the relevant features for retrieval vary depending on the user's intent. Conditional representation learning enables the model to generate image representations that are tailored to the given criterion conveniently and quickly, improving retrieval efficiency and user satisfaction.

We propose conditional representation learning and offer a computationally efficient and highly generalizable solution. This is exactly where our novelty lies.

Besides, our proposed method is a simple, plug-and-play approach. Nevertheless, we believe this is not a weakness but rather a strength. Furthermore, we provide a basis transformation perspective to justify the technical soundness of our method.

Weakness 1.2: Comparison to zero-shot classification

While equation 4 is similar in computational form to zero-shot classification, our method is fundamentally different in the following two aspects:

1) Different tasks: Zero-shot classification directly outputs a predicted class label, solving the classification task. By comparison, our method serves as a plug-and-play module for the conditional representation learning task.

2) Different priors: Zero-shot classification relies on the availability of class names as prior knowledge, while our method does not require them. Obtaining such priors is impractical in domains featuring a large number of classes. For example, in the fashion retrieval task(Section 4.2.2), the DeepFashion dataset we adopt consists of 1000 fine-grained classes. What's more, in real-world applications, it's even harder to make class names as priors since they are dynamic.

Weakness 1.3: Performance

We respectfully note that the performance degradation does not occur as "often" as the comment suggested.

In fact, across all four tasks, our method consistently outperforms the base model and existing baselines. Specifically, for the linear probe task (Table 1), CRL obtains a performance boost of the original CLIP representation of over 21% relative. For the clustering task (Table 2), CRL obtains a performance boost of the SOTA method (Multi-Map) of over 53%. For the customized similarity retrieval task (Table 3), CRL achieves a gain of 13% compared to the SOTA method (Combiner). For the fashion retrieval task (Table 4), CRL surpasses the best competitive method (RPF) by 10%. The few cases of degradation mentioned in the comment are isolated exceptions rather than the norm.

Weakness 2: Benchmark

Our work falls under the scope of representation learning. Therefore, we selected downstream tasks that are closely tied to this area, specifically, classification and retrieval. We included four such sub-tasks, which we believe are sufficient to validate the effectiveness of our approach.

Nevertheless, we truly value the suggestion of applying our method to the VQA problem. Since we are not familiar with this field and previous representation learning works seldom select the VQA problem as the downstream task, we wonder how to apply our method to it.

By the way, we believe this is feasible, as the dataset used in the customized similarity retrieval task (Section 4.2.1) is derived from the COCO dataset. We'll leave this as a promising direction for future exploration.

Weakness 3: Baseline

Thanks for the insightful suggestion. Before the rebuttal, we indeed only considered applying our method based on the CLIP-like architecture. Following your suggestion, we apply our method to BLIP2 and find that it even outperforms CLIP, as the following table shows. This further demonstrates the generalizability of our work, and we will continue exploring broader applicability in future research.

Moreover, except for the linear probe task, the other three tasks are also compared against methods specifically designed for those tasks. To be specific, for the clustering task (Section 4.1.2), the baselines include three specifically designed methods: CC, SCAN, and Multi-Map. For the customized similarity retrieval task (Section 4.2.1), the baselines include five representative methods: Pic2Word, SEARLE, LinCIR, CIG, and Combiner. For the fashion retrieval task (Section 4.2.2), we compare against five competitive baselines: Triplet, CSN, ASEN, ASEN++, and RPF.

Weakness 4: Discussion on limitations

Despite CRL's generalizability across different criteria, it may not outperform clustering methods like CC and SCAN under the universal criterion "shape". This is likely because these methods employ specially targeted designs for clustering under this criterion.

We will add the above results and discussions in the next version.

Finally, we would like to thank you for your time and effort once again, and sincerely hope our response could address your concerns.

Dear reviewer xrw3,

We hope this message finds you well. As the rebuttal deadline is approaching, we wanted to kindly check in to see if you might have any comments or feedback for us to address. Your insights are important to us, and we would like to make sure we have sufficient time to incorporate your suggestions into our response. Thank you very much for your time and effort.

We sincerely thank you for the time and effort spent reviewing our paper. Nevertheless, we respectfully believe that some of the concerns raised may be due to misunderstandings, and we would like to clarify these points in detail below.

Weakness 1.1: Related work

First of all, we must point out that the core of our work, CRL (conditional representation learning), aims to obtain conditional representations tailored to the user-specified criterion without any supervision signal.

According to this, we choose "representation learning" and "conditional similarity" as our related work. The former refers to traditional methods that aim to learn general feature representations, while the latter focuses on newly emerging areas that learn conditional similarity conditioned on specific criteria.

The relevant works you mentioned, MMRL (fine-tuning the model structure for image classification) and SwapPrompt (adapting the VLM prompt for image classification), are two works that aim to learn representations under the general criterion (shape or category). We'll add them to the "representation learning" section in related works.

Weakness 1.2: Methodology novelty

We disagree with the comment "the methodology of TextSpan and IC|TC is almost identical to the one proposed by the authors". Below is an intuitive comparison among these three works:

1) Different tasks: CRL aims to obtain conditional representations under specified criteria for multiple tasks. TextSpan seeks to analyze the role of each head in the CLIP model. IC|TC is designed for the clustering task.

2) Different techniques: CRL utilizes the descriptive texts as the semantic basis, and then projects images into this conditional feature space to obtain conditional representations. TextSpan iteratively selects a set of texts that maximizes the variance of the head's output features, using these texts to interpret the corresponding head. IC|TC queries the LLM to get the cluster affiliation with three steps.

It's clear to see that CRL is substantially different from TextSpan and IC|TC.

Weakness 2.1: Dataset scale

To the best of our knowledge, previous multi-criteria works typically focused on a single downstream task. So in this paper, we integrate existing multi-criteria datasets from multiple domains. We acknowledge that, as an emerging field, this area currently lacks abundant high-quality datasets. However, we would like to point out that not all dataset scales are small, as the DeepFashion dataset contains over 270k images (in the 4.2.2 section).

We hope our work can inspire broader interest in this promising research direction. Moving forward, we also intend to develop larger-scale datasets to support continued progress.

Weakness 2.2: Missing confidence intervals

We present the confidence intervals (at the confidence level of 0.95) for both the customized clustering and linear probe tasks across 20 trials. The experiments in the rebuttal are all conducted under the same setting.

The table below reveals that the random seed has minimal impact on the performance of our method.

We'll complement tables 1, 2, 3, and 4 in the main text for the next version.

Weakness 3.1: LLM ablation

Thanks for the suggestion, and we provide thorough ablation studies below.

As for the selection of LLMs, we would like to clarify that our method does not particularly rely on any specific LLM. To be specific, we use the same prompt to query four mainstream LLMs, GPT-4o, Deepseek-v3, Gemini 2.5, and Claude 4.

The result demonstrates that all four LLMs achieve comparable and consistent performance.

Weakness 3.2: LLM prompt ablation

As for the LLM prompt, we require it to include the [criterion]. We devise below 5 different templates:

1) Generate common expressions to describe the [criterion].

2) List a wide variety of typical phrases used to characterize the [criterion].

3) Enumerate familiar terms or expressions people often use when referring to the [criterion].

4) Identify and list expressions frequently used to convey the concept of the [criterion].

5) How do people usually talk about the [criterion]?

The table above shows that different LLM prompts can yield close performance improvement, indicating that our method is robust against the LLM prompt.

Weakness 3.3: VLM prompt ablation

As for the VLM prompt, we require it to contain the [criterion] and the generated [text] by the LLM. We also devise below 5 different templates:

1) objects with the [criterion] of [text]

2) a photo with the [criterion] of [text]

3) itap with the [criterion] of [text]

4) art with the [criterion] of [text]

5) a cartoon with the [criterion] of [text]

As suggested in the above table, our method remains stable across different VLM prompts.

Weakness 3.4: Temperature ablation

As for the LLM temperature t, we set t to 0, 0.5, 1 and 1.5 to obtain the text basis, respectively. The temperature ranges from 0 to 2, with higher values introducing more variability and randomness in the LLM's output (When the temperature approaches 2, the generated content becomes almost entirely random, so we did not include this setting in our experiments).

The experimental results also validate the robustness of our method to the temperature parameter.

Weakness 3.5: Redundancy ablation

As for the redundancy, we analyze the number of generated words in Figure 6 in the main text. The result shows that our method maintains stable performance, indicating robustness to redundancy.

Weakness 4.1: Baseline

It appears that the scope of our baseline comparisons may have been overlooked. Our proposed method is built upon the representations of base models. Consequently, it is intuitive to observe the effect of representation transformation when compared directly to these representations.

However, we don't limit our comparisons to the base models. In fact, except for the linear probe task (which is a direct evaluation of the transformation), the other three tasks are also compared against methods specifically designed for those tasks.

To be specific, for the clustering task (Section 4.1.2), the baselines include three specifically designed methods: CC, SCAN, and Multi-Map. For the customized similarity retrieval task (Section 4.2.1), the baselines include five representative methods: Pic2Word, SEARLE, LinCIR, CIG, and Combiner. For the fashion retrieval task (Section 4.2.2), we compare against five competitive baselines: Triplet, CSN, ASEN, ASEN++, and RPF.

Weakness 4.2 Performance

We think this comment may be overly strict. In fact, our method, CRL, shows significant improvements over both the base models and existing baselines across all four tasks. Specifically, for the linear probe task, CRL obtains a performance boost of the original CLIP representation of over 21% relative. For the clustering task, CRL obtains a performance boost of the SOTA method (Multi-Map) of over 53%. For the customized similarity retrieval task, CRL achieves a gain of 13% compared to the SOTA method (Combiner). For the fashion retrieval task, CRL surpasses the best competitive method (RPF) by 10%.In other words, the cases of performance degradation are isolated exceptions rather than indicative of overall trends. Moreover, we would like to clarify that for the “count” criterion on the clevr4-10k dataset in Table 1, there is no actual performance degradation.

We will add the above results and discussions in the next version.

Finally, we sincerely hope that our response will be carefully considered and lead to a more favorable assessment. Thank you for the time spent reviewing our paper again.

Dear reviewer iGLB,

We are reaching out with a gentle reminder as the rebuttal submission deadline is drawing near. We noticed that your comments have not yet been posted, and we would be very grateful if you could share any feedback or concerns in the remaining time. Your input is crucial for us to respond meaningfully. We sincerely appreciate your efforts and look forward to your thoughts.