Customized Multiple Clustering via Multi-Modal Subspace Proxy Learning

W1: I wonder if this is another form of two-stage task.

Thanks for your invaluable feedback. We will make it clear in the revision as follows. A two-stage process used by previous methods separates the representation learning and clustering entirely, where the representation learning is fully completed at first. This separation, however, can lead to sub-optimal clustering results, as the learned representations may not be fully aligned with the clustering objective. Instead, we obtain both the proxy word and the clustering alternatively. Concretely, we first learn the proxy word in a user-preferred subspace (i.e., Eqn. (5) in the submission). Then, we fix the proxy word and open the image encoder to obtain better image representations considering the clustering objective (i.e., the clustering loss at line 240 in the submission). These two steps are repeated until convergence. Therefore, these two components can improve each other iteratively, which shows better performance than the separated two-stage strategy in previous methods.

---

W2: The description of Clustering Loss is not very clear in Section 3.4, how to determine that samples belong to the same class? By pseudo-label? Where did the pseudo-label come from?

Thanks for pointing this out. During the clustering, the proxy word is fixed, which can be used to obtain the pseudo-labels to determine the inter-cluster and intra-cluster relationships. Concretely, an offline clustering algorithm (e.g., k-means) can be applied to the currently fixed proxy words to group the samples as the pseudo-labels. We will clarify it in the subsequent versions.

---

W3: Does the direct use of a pre-trained large language model introduce a priori information about the category, which can lead to unsupervised scenarios being corrupted?

Thanks for your insightful comments. It should be noted that although the category information released by the LLM can help determine a better subspace for the proxy learning, it is hard to cover all ground-truth labels, especially for user-specific domains as in this work. The provided categories by the LLM can be too narrowed, too expansive, or without any overlapping. Therefore, the corresponding categories existing in the data are unknown or uncertain. Moreover, the exact label for each instance is still unknown as in most unsupervised scenarios. We will make it clear in the revision.

W1: Several associations.

Thanks for your insightful comments. Multi-Sub works by learning in the user-preferred subspace. Therefore, it is theoretically unlikely that the learned representations are completely opposite to the user's demand under such aligned subspace. We will make it clear in the revision.

To evaluate how Multi-Sub adapts to new user demands not originally provided in the ground-truth of the dataset, we conducted an additional experiment using the Fruit dataset. Specifically, we introduced a new demand based on the shape of the fruits, with the prompt set as "fruit with the shape of *". We categorized the fruits into two shapes, that is, round and elongated. Although this specific demand may not be common in practical applications, it serves as an exploratory experiment to test the adaptability of our method.

The following results demonstrate that Multi-Sub successfully adapted to the new user demand of shape. The model learned to align the textual descriptions of shapes with the visual features, resulting in a clustering under the new subspace of shape. Thanks for the suggestion and we will include this experiment in the revision.

---

W2: Figure 2.

Thanks for pointing this out and we will carefully revise Figure 2 in the revision.

---

W3: Clustering Loss.

Thanks for the suggestion and we will make the following clear in the revision. Previous methods often use a two-stage process that they separate representation learning and clustering to simplify the optimization process. This separation, however, can lead to sub-optimal clustering results, as the learned representations may not be fully aligned with the clustering objective. Instead, we obtain both the proxy word and the clustering alternatively and simultaneously. Concretely, we first learn the proxy word in a user-preferred subspace (i.e., Eqn. (5) in the submission). Then, we fix the proxy word and open the image encoder to obtain better image representations considering the clustering objective (i.e., the clustering loss at line 240 in the submission). These two steps are repeated alternatively until convergence.

---

W4: Datasets.

Thanks for the suggestion. We will make the following descriptions clear in the revision.

1. Stanford Cars: This dataset contains 1,200 images of cars annotated with labels for both color and type (e.g., sedan, SUV).
2. Card: This dataset includes 8,029 images of playing cards with labels for rank (Ace, King, Queen, etc.) and suits (clubs, diamonds, hearts, spades).
3. CMUface: This dataset consists of 640 facial images with annotations for pose, identity, glasses, and emotions.
4. Flowers: This dataset includes 1,600 flower images labeled by color and species (e.g., iris, aster).
5. Fruit: This dataset comprises 105 images of fruits with labels for species (apples, bananas, grapes) and color (green, red, yellow).
6. Fruit360: Similar to the Fruit dataset, Fruit360 contains 4,856 images of various fruits with detailed annotations for species and color.
7. CIFAR-10: This dataset is structured to have 60,000 images clustered based on type (transportation, animals) and environment (land, air, water).

The data size, handcrafted features, and clusters for all datasets we used are also summarized in the following table. It is worth noting that, in our experiments, we apply both traditional and deep learning baselines. Traditional methods rely on hand-crafted features, while deep learning methods directly utilize the original images as input.

---

Question.

We do have some additional experiments as in the one-page PDF that we will add in the revision. Those experiments include:

• Clustering performance based on shape demand on the Fruit dataset, demonstrating the adaptability of Multi-Sub for new demands not existing in ground-truth clusterings provided by the data.
• Sensitivity analysis of the balancing factor $\lambda$ on the CIFAR-10 dataset, indicating a best value of 0.5, which is expected since when $\lambda$ is too low, the model focuses too much on maximizing the distances between different clusters, which can lead to less cohesive clusters. Conversely, when $\lambda$ is too high, the model emphasizes compactness within clusters at the expense of inter-cluster separation, leading to less distinct clusters. Therefore, 0.5 is used for all datasets.
• Quantifying feature space differences generated by different text encoders using the Maximum Mean Discrepancy (MMD) metric, showing that even with simple text prompts in our work, different pre-trained text encoders vary in their abilities.

W1: The contributions of the paper.

Thank you for the suggestion. We will carefully emphasize our contribution in the revision as follows:

Given only a high-level user interest in an unsupervised scenario without any class labels or names, we cannot directly apply CLIP. Instead, we must learn the unknown proxy word in a continuous space, which is challenging. The subspace approach proposed in this work aids effective learning. More importantly, our proposal can simultaneously optimize the proxy word and the clustering result. We will also emphasize this in our framework figure in the revision.

---

W2: The baseline methods chosen for comparison.

Thank you for pointing this out. We will provide detailed descriptions in the revision as follows:

1. MSC [2] is a traditional multiple clustering method that uses hand-crafted features to automatically find different feature subspaces for different clusterings.
2. MCV [1] leverages multiple pre-trained feature extractors as different views of the same data.
3. ENRC [3] integrates auto-encoder and clustering objective to generate different clusterings.
4. iMClusts [4] is a deep multiple clustering method that leverages the expressive representational power of deep autoencoders and multi-head attention to generate multiple salient embedding matrices and multiple clusterings therein.
5. AugDMC [6] leverages data augmentations to automatically extract features related to different aspects of the data using a self-supervised prototype-based representation learning method.
6. DDMC [5] combines disentangled representation learning with a variational Expectation-Maximization (EM) framework.
7. Multi-MaP [7] relies on a contrastive user-defined concept to learn a proxy better tailored to a user's interest.

---

W3: Broader impacts.

Thank you for the suggestion. We will add the discussion that our proposal has potential in various applications like personalized marketing. For example, it can enhance advertisement effectiveness by tailoring clustering results to business preferences.

---

Question: How were the multiple clustering ground truth values in the dataset obtained?

All multiple clustering datasets used in our experiments are public and come with pre-defined ground-truth labels in different aspects for evaluation that have been widely used. For datasets like Stanford Cars and CIFAR-10, the clustering ground truth labels are derived from their existing class labels. Below are the details of the datasets:

1. Stanford Cars: This dataset contains 1,200 images of cars annotated with labels for both color and type (e.g., sedan, SUV).

2. Card: This dataset includes 8,029 images of playing cards with labels for rank (Ace, King, Queen, etc.) and suits (clubs, diamonds, hearts, spades).

3. CMUface: This dataset consists of 640 facial images with annotations for pose, identity, glasses, and emotions.

4. Flowers: This dataset includes 1,600 flower images labeled by color and species (e.g., iris, aster).

5. Fruit: This dataset comprises 105 images of fruits with labels for species (apples, bananas, grapes) and color (green, red, yellow).

6. Fruit360: Similar to the Fruit dataset, Fruit360 contains 4,856 images of various fruits with detailed annotations for species and color.

7. CIFAR-10: This dataset is structured to have 60,000 images clustered based on type (transportation, animals) and environment (land, air, water).

The data size, handcrafted features, and clusters for all datasets we used are also summarized in the table. It is worth noting that, in our experiments, we apply both traditional and deep learning baselines. Traditional methods rely on hand-crafted features, while deep learning methods directly utilize the original images as input.

---

References

[1]. J. Guérin and B. Boots. Improving image clustering with multiple pretrained cnn feature extractors.In British Machine Vision Conference 2018, BMVC 2018, 2018.

[2]. J. Hu, Q. Qian, J. Pei, R. Jin, and S. Zhu. Finding multiple stable clusterings. Knowledge and Information Systems, 2017.

[3]. L. Miklautz, D. Mautz, M. C. Altinigneli, C. Böhm, and C. Plant. Deep embedded non-redundant clustering. In Proceedings of the AAAI conference on artificial intelligence, 2020.

[4]. L. Ren, G. Yu, J. Wang, L. Liu, C. Domeniconi, and X. Zhang. A diversified attention model for interpretable multiple clusterings. IEEE Transactions on Knowledge and Data Engineering, 2022.

[5]. J. Yao and J. Hu. Dual-disentangled deep multiple clustering. In Proceedings of the 2024 SIAM International Conference on Data Mining (SDM), 2024.

[6]. J. Yao, E. Liu, M. Rashid, and J. Hu. Augdmc: Data augmentation guided deep multiple clustering. In INNS DLIA@IJCNN, 2023.

[7]. J. Yao, Q. Qian, and J. Hu. Multi-modal proxy learning towards personalized visual multiple clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2024.

W1: A more detailed analysis and discussion on the sensitivity of the method to these parameters.

We greatly appreciate your suggestion. To show the sensitivity of the balancing factor $\lambda$ that is the only hyper-parameter in our proposal, the experiments were conducted on CIFAR-10. We varied the value of $\lambda$ from 0.1 to 1.0 in increments of 0.1 to observe its effect on the model's performance. As shown in Figure 1 of the one-page PDF, we can observe that different values of $\lambda$ can work with our method and the optimal performance for "Type" & "Environment" clustering is achieved when $\lambda$ is set to 0.5. When $\lambda$ is too low, the model focuses too much on maximizing the distances between different clusters, which can lead to less cohesive clusters. Conversely, when $\lambda$ is too high, the model emphasizes compactness within clusters at the expense of inter-cluster separation, leading to less distinct clusters. Therefore, we set $\lambda$ to be 0.5 for all datasets, which confirms the robustness of our method.

---

W2: Whether regularization techniques were used to address overfitting?

We would like to clarify that the primary objective of our method is clustering rather than supervised learning where overfitting is a more prevalent issue. Specifically, clustering focuses on finding inherent patterns and structures in the data rather than fitting to specific target labels. Therefore, the regularization in clustering helps constrain the data subspace to observe data groups rather than alleviate the overfitting of a model. In this work, we have subspace under a user's preference in the proposed method for regularization. We will make this clear in the revision.

---

W3: There are significant performance differences when using different encoders, this phenomenon could be explored in depth.

Thanks for your insightful comments. We conducted an additional analysis using the Maximum Mean Discrepancy (MMD) metric to quantify the differences in the feature spaces generated by different text encoders (i.e., CLIP, ALIGN, and BLIP).

The MMD results indicate that although our text prompts are simple, the feature spaces generated by different text encoders exhibit significant distributional differences.

The effectiveness of a text encoder can vary depending on the specific clustering task. For example, ALIGN tends to excel in tasks with more abstract attributes, such as colors and emotions, while CLIP shows strong performance in identity-related tasks. This variability underscores the importance of selecting an appropriate text encoder based on the specific application requirements. The difference between text encoders may come from the different corresponding pre-training tasks and this will be an interesting future direction.