Stabilizing Sample Similarity in Representation via Mitigating Random Consistency

Dear Reviewer WSCE,

     We are very grateful for your valuable comments and questions. The responses are as follows:

Response to Weaknesses 1:

     For the image dataset, we solely employ VGG and MoCo v3 for feature extraction. The specific methodology is detailed in Section 7.2 of the main text and Appendix 3.0.1. In this modification, we trained more sophisticated networks (ViT and CNN) using our loss function. The experimental results and detailed parameter configurations are as follows.

     In this study, we conduct experiments utilizing a hardware configuration that includes an Intel (R) Core (TM) i7-14700F CPU, 16GB of RAM, and an NVIDIA GeForce RTX 4060 GPU. The experiments are carried out on a Windows operating system, with Python 3.10 serving as the programming language and the PyTorch 2.4 library employed for model development and training.

     For the Vision Transformer (ViT) [1] model training, we use a stochastic gradient descent (SGD) optimizer with a learning rate set to 0.001. The batch size is set to 64, and the model is trained for a total of 100 epochs. The dataset is partitioned into training and testing subsets with a ratio of 7:3.

     Similarly, the ConvNet [2] model employs the AdamW optimizer, with the learning rate set to 0.004. This model also uses a batch size of 64 and undergoes training for 100 epochs. In alignment with the ViT model, the dataset is divided into training and testing sets, preserving the same 7:3 ratio.

     Performance evaluation of both models is conducted using accuracy and F-measure as the primary metrics. Analysis of the results, as presented in the table, our method shows better performance over existing approaches. This indicates the effectiveness of our method.

[1] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations, 2021.

[2] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, et al. A ConvNet for the 2020s. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11966–11976, 2022.

Response to Weaknesses 2:

     Thank you for your suggestions. We have added a runtime comparison, as shown in the table below. In this study, we conduct experiments utilizing a hardware configuration that includes an Intel (R) Core (TM) i7-12700F CPU, 32GB of RAM, and an NVIDIA GeForce RTX 3060 Ti GPU. For the first 20 benchmark datasets, the table below presents the total time consumption (in seconds) for both training and testing. For image datasets, the recorded values represent the training and prediction time (in seconds) on the fully connected network shown in Figure 3, after feature extraction using either MoCo v3 or VGG. From the table, it can be observed that our method does not significantly increase computational time.

Dear Reviewer 72cG,

We are very grateful for your valuable comments and questions. The responses are as follows:

Response to Claims And Evidence: The proposed loss function serves as a universal quality measure for similarity matrices, which are foundational elements across various learning paradigms. Its generalized formulation enables applications in training deep network, metric learning, kernel learning and so on. To demonstrate the broad applicability, we applied it to train advanced network architectures (ViT and CNN). The experimental results and detailed settings are listed in our response to Reviewer WSCE Weakness 1.

Response to Experimental Designs Or Analyses: By incorporating the random consistency mitigation mechanism (Eq. 5), we have enhanced the homogeneity and unbiasedness of the loss function, thereby improving the model's generalization capability. Significance tests demonstrate that this improvement maintains stable performance advantages across different data partitions, confirming that the enhancement is not attributable to random factors.

Response to Weaknesses 1: This is an excellent suggestion. To enhance completeness and readability, we will add both a conceptual introduction to RC and an expanded literature review in the appendix of the next edition. For further details, please see Question 3.

Response to Weaknesses 2: This is a constructive suggestion. Indeed, the opening paragraph of Section 6 has some deficiencies in clarifying the motivation and scope behind our choice of network architecture. In fact, to validate the effectiveness of the loss function, we deliberately opted the proposed loss function for the most basic fully connected network structure. As mentioned earlier, our proposed loss function has relatively broad applicability. In the revised version, we will explicitly elaborate on the scope and adaptability of the method in section 6.

Response to Question 1: The dependency of existing methods on uninformative matrices can be derived from Eq. 1, since they measure the information content by computing the distance from uninformative matrices (e.g., identity matrix or full one matrix).

Response to Question 2: The motivation of this paper is to propose a loss function ($d_{PSED}$) for measuring the information content of similarity matrices when ground-truth labels are available. The shortcoming of $d_{infor}(\mathbf{K})$ lies in its fundamental inability to properly assess the block-diagonal structure of similarity matrices, as evidenced by Property 1. To address this issue, $d_{PSED}$ introduces the ground-truth adjacency matrix as the reference standard, thereby enabling quantitative assessment of how closely the similarity matrix approximates the ideal block structure. The limitation of $d_{SED}(\mathbf{K})$ manifests as differential biases toward identity matrix $I$ versus all-ones matrix $J$ across varying class distributions. Through mitigating random consistency, we obtain $d_{PSED}$, a weighted element-wise dissimilarity metric for similarity matrices that achieves unbiased evaluation of both $I$ and $J$. The $d_{PSED}$ mitigates the random consistency issue by subtracting the expectation of $d_{SED}(\mathbf{K})$ under random label permutations (Eq.5).

Response to Question 3: Consistency metrics measure the agreement between two random variables, while random consistency(RC) refers to spurious agreement arising purely from randomness[1]. A canonical manifestation of RC occurs when examinees achieve measurable test scores solely via random response patterns. The mechanisms by which RC harms the learning process include evaluation distortion, optimization misguidance, and generalization barriers. Failure to deduct the RC baseline may lead to overestimating the model’s actual consistency performance (e.g., an original consistency score of 0.6 vs. a random baseline of 0.2 means the true effective consistency should be 0.4). When loss functions include RC without proper correction, they can induce optimization bias, causing algorithms to spuriously improve consistency metrics by overfitting to noise [2] or data bias[3,4] instead of learning genuine data patterns. These would consequently impair the model's generalization ability.

[1]Wang J, Qian Y, Li F, et al. Generalization performance of pure accuracy and its application in selective ensemble learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 45(2): 1798-1816.

[2]Wang J, Qian Y, Li F. Learning with mitigating random consistency from the accuracy measure. Machine Learning, 2020, 109: 2247-2281.

[3]Li J, Qian Y, Wang J, et al. PHSIC against random consistency and its application in causal inference. IJCAI, 2024, 5(10): 15-20.

[4]Vinh N X, Epps J, Bailey J. Information theoretic measures for clusterings comparison: Variants, Properties, Normalization and Correction for Chance. Journal of Machine Learning Research, 2010, 11: 2837–2854.

Dear Reviewer GtCV,

     We are very grateful for your valuable comments and questions. The responses are as follows:

Response to Methods And Evaluation Criteria:

     For the image dataset, we firstly employed feature extractors such as MoCo v3 or VGG for processing, with downstream classification processing handled by fully connected layers. The specific methodology is detailed in Section 7.2 of the main text and Appendix 3.0.1. We note that there may be a typesetting oversight in Figure 3, where images were mistakenly labeled as direct inputs. This issue will be corrected in the revised version. Besides, in this modification, we implement our loss function on a more sophisticated network architecture(ViT and CNN). The experimental results and detailed settings are listed in our response to Reviewer WSCE Weakness 1.

Response to Experimental Designs Or Analyses:

     We adopted random splitting to generate the training and test sets, as described in Appendix 3.0.1.

Response to Other Comments Or Suggestion 1:

     The consistency metric quantifies the degree of agreement or alignment between different variables. Random Consistency (RC) refers to agreement or similarity that occurs purely by chance, not because of any true underlying pattern or relationship. When the sample size is limited or the noise level is high, the random consistency (RC) between the classifier's predicted labels and the true labels increases [1]. For more details on RC, please refer to Question 3 of Reviewer GtCV.

[1] Jieting Wang, Yuhua Qian, Feijiang Li. Learning with Mitigating Random Consistency from the Accuracy Measure, Machine Learning, 2020, 109: 2247-2281

Response to Other Comments Or Suggestion 2-4:

     $K$, $I$, and $J$ are defined in Equation (1), though we will include their definitions after each subsequent formula. Thank you for your suggestion. Thank you for pointing these out and we will modify them accordingly.

Response to Question 1:

     CE-PSED refers to the summation of CE and PSED. Thank you for pointing this and we will add this in the final version.

Response to Question 2:

     This is indeed a profound question. As illustrated in Figures 7 and 9 in the appendix, the t-SNE visualization of cross-entropy (CE) for the MPEG and Caltech-101 datasets fails to reveal clear inter-class structures. By introducing the block-diagonal constraint on the similarity matrix—that is, incorporating PSED as part of the loss function—we can enhance inter-class separation in data representation, thereby improving model performance, shown as Table 7 in Appendix. We will add this explanation in the final version.

Response to Question 3:

     For the image datasets, we firstly employed MoCo v3 or VGG for feature extracting, with downstream classification processing handled by fully connected layers. The specific methodology is detailed in Section 7.2 of the main text and Appendix 3.0.1.