DenoiseRep: Denoising Model for Representation Learning

Q1: What constitutes this noise?

A1: We appreciate your detailed review. Experimental results of Table 3 (FEFDUF) could empirically demonstrated the hypothesis of "the features obtained by backbone extraction are noisy". FEFDUF includes a well-trained person ReID model, and a denoise model which take feature of the ReID model as input and predict its noise. During the training stage, the ReID model is always frozen, and denoise model is trained under normal process in DDPM (i.e. adding a gaussian noise to feature, taking the nosied feature as input and predicting the original gaussian noise). During the inference stage, given images, we first extract features A with the well-trained ReID model, then put the features into the denoise module to predict its noise A'. Finally, we found that A-A' performs stably better than A. We will add the analysis above to the revised manuscript.  

Q2: The authors should evaluate the proposed method on these datasets instead of conventional ones.

A2: Thank you for the valuable suggestion. We validate our proposed FEFDFA on Occluded-Duke [1], Occluded-ReID [2] and Partial-ReID [3]. These datasets include person images occluded by misc obstacles such as vehicles, trees and so on, making them more challenging. We will add the experimental results to the revised manuscript. The experimental results are as follows, using mAP as the evaluation metric:

Q3: The authors lack an analysis of the model's complexity, including the number of parameters and training time.

A3: Thanks for your valuable feedback. Our proposed method takes LITTLE extra parameters and training time. The reason is the denoising layers employ simple linear layers, they are very light compared with heavy vision backbone (e.g. ViT, ResNet). For example, the parameter of ViT-Small is 22,313,320, while the sum of the parameters of the denoising module is only 3,538,944, an additional increase of about 15.8%. The detailed experimental results are as follows. Please note that these are training-time parameters, our proposed method takes no extra latency during the test stage.

Q4: The performance gain are minimal.

A4: Thanks for your valuable feedback. Our proposed method is test-time computation-free (FREE LAUNCH!) and scalable to various baseline and downstream tasks, which is proven to be effective in CNN series (e.g. ResNet50), Transformer series (e.g. ViT, Vmanba), Person Re-ID (MSMT, Duke), Vehicle Re-ID (vehicReID), Image Classification (ImageNet, CUB). The latest experimental results on CLIP-REID show better improvement on MSMT (CLIP-REID: 75.8% v.s. CLIP-REID+OURS: 76.5% for mAP, see the table below). The code will be released after the manuscript is accepted. We will add the experimental results to the revised manuscript.  

Q5: The authors should verify the effectiveness of the proposed method using the visual encoder in CLIP as the backbone.

A5: Thanks for your insightful suggestion. CLIP is a very strong vision-text encoder, which is trained with 400 million data. Beyond CLIP, CLIP-ReID pioneeringly adapts CLIP, a zero-shot classifier, to ReID, a fine-grained image retrieval task. Our proposed FEFDFA is model-free, thus it can be easily applied to CLIP-ReID without any modification. The experimental results on CLIP-REID show stable improvement on MSMT (CLIP-REID: 75.8% v.s. CLIP-REID+OURS: 76.5% for mAP). The code will be released if the manuscript could be accepted. We will add the experimental results to the revised manuscript. The experimental results are as follows, with mAP as the evaluation metric:

[1] Jiaxu Miao, Yu Wu, Ping Liu, Yuhang Ding, and Yi Yang. Pose-guided feature alignment for occluded person re-identification. In ICCV, pages 542–551, 2019. 1, 2, 6

[2] Jiaxuan Zhuo, Zeyu Chen, Jianhuang Lai, and Guangcong Wang. Occluded person re-identification. In ICME, pages 1–6. IEEE, 2018. 2, 6

[3] Wei-Shi Zheng, Xiang Li, Tao Xiang, Shengcai Liao, Jianhuang Lai, and Shaogang Gong. Partial person reidentification. In ICCV, pages 4678–4686, 2015. 6

The authors slove all my quentions. Overall, the motivation of this paper is clear, the idea is novel, the proposed is simple yet effective, and the writting is satifactory.

Hi, dear reviewer, we hope our responses have solved your concerns. If you still have some concerns, please feel free to raise them here and we will respond as soon as possible.

Q1: Clarify the difference between the proposed FEFDFA and reparameterization.

A1: Thank you for your insightful comment. We appreciate the opportunity to clarify the differences between our "Feature Extraction and Feature Denoising Fusion Algorithm" (FEFDFA) and reparameterization.

• Reparameterization: Reparameterization typically involves restructuring the parameters of a model to facilitate more efficient training. It is widely used in variational autoencoders (VAEs) to allow for gradient-based optimization of stochastic variables.
• Feature Extraction and Feature Denoising Fusion Algorithm (FEFDFA): Our FEFDFA is designed to integrate feature extraction and denoising within the same backbone. Each embedding layer in the backbone serves a dual purpose: extracting features and performing denoising simultaneously. By deriving the formula for diffusion model sampling, the denoising layer parameters are fused with the feature layer parameters, so that the model has no additional time cost in the inference stage.  

Q2: It will be better if peformance on CNN series are shown.

A2: Thanks for your valuable feedback. Our proposed method is scalable to various baseline (including CNN, see Table 5 and line seven of Table 6 for details) and downstream tasks, including CNN series (e.g. ResNet50), Transformer series (e.g. ViT, Vmanba), Person Re-ID (MSMT, Duke), Vehicle Re-ID (vehicReID), Image Classification (ImageNet, CUB). The experimental results in Section 4.4 of the main text and Appendix C demonstrate that our method significantly improves these tasks.  

Q3: Representation learning applications on more downstream tasks.

A3: Thanks for your valuable suggestion. Representation learning is a broad and foundational concept with potential applications across various downstream tasks. Due to the limited time, we extend our proposed FEFDFA to the object detection task with Mask-RCNN[2] as a baseline and COCO[1] as a benchmark. Experimental results show that ours carries stable improvements of 1.6%-1.1%. Please see the table below for details. We will extend ours to more downstream tasks including segmentation and generation, in future.

Q4: The hyper-parameter analysis is missed.

A4: Thanks for your reminder. We analyze hyperparameters in the model, including important parameters such as $\beta_t$ (Line178), and diffusion step size T (Line73). The results of the hyper-parameter analysis will be included in the revised manuscript, along with discussions on how different settings affect the model's performance. We use ViT-small as the backbone and conducted experiments on the DukeMTMC dataset with mAP as the evaluation metric. The experimental results are as follows:

Q5:What if the baseline methods are trained under the same steps?

A5: Thank you for your question. We conduct experiments to address this concern. We train the baseline methods using the same number of steps as our proposed method. However, we do not observe any significant performance improvements in the baseline methods. We will include these experimental results and detailed analyses in the revised manuscript to support this explanation. Thank you for your valuable feedback. The experimental results are as follows：

where the original baseline trained for 120 epochs. Therefore, when the epoch is 120, our method did not participate in fine-tuning. When the epoch is 160, our method fine-tunes for another 40 epochs based on the pre-trained parameters of the baseline, and so on.

[1] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollar, and C. L. Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014. 2, 5

[2] He, Kaiming, et al. "Mask r-cnn." Proceedings of the IEEE international conference on computer vision. 2017.

Q1: The authors need to enrich the Related Work.

A1: We thank the valuable suggestions. Our proposed DenoiseReID is different from the related works [1-3]. The related works [1-3] apply the itermediate layer features of an existing pre-trained diffusion model to improve downstream task. Ours applies the denoise algorithm to any existing pre-trained downstream model (e.g. Person ReID of Table 2, Vehicle ReID of table 5, Image Classification of Table 6). Compared with the related works [1-3], ours is more scalable (suitable to more downstream tasks) and light-weight (computation-free). We will carefully review and add them to the revised manuscript.  

Q2: Line144-146, no evidence provided to support the proposed hypothesis.

A2: Thanks for the kind comment. Experimental results of Table 3 (FEFDUF) could empirically demonstrate the hypothesis of "the features obtained by backbone extraction are noisy". FEFDUF includes a well-trained person ReID model, and a denoise model which takes the feature of the ReID model as input and predicts its noise. During the training stage, the ReID model is always frozen, and the denoise model is trained like DDPM (i.e. adding a gaussian noise to feature, taking the nosied feature as input and predicting the original gaussian noise). During the inference stage, given images, we first extract feature A with the well-trained ReID model, then put the feature into the denoise module to predict its noise A'. Finally, we found that A-A' performs better than A. The training process DO NOT use any label because DDPM is an unsupervised manner. This experiment shows that simple denoising features contribute to improvement, partially supporting the view that "the features obtained by backbone extraction are noisy". We will add the analysis above to the revised manuscript.  

Q3: The training loss in Eq. (11) should be the sum of the MSE losses of each denoising layer.

A3: Thanks for the valuable reminder. This is a writing error. In the code implementation, the loss during the training of the denoising module is the sum of the MSE losses of each denoising layer. The code will be released if the manuscript could be accepted. We will polish the Eq. (12) to be:

Q4: The statement is inconsistent.

A4: Thanks for your comments. There are THREE questions. Let's discuss them ONE BY ONE:

• (1) "we freeze the original parameters and only train the FEFDFA." This is our basic training setting. It only trains the parameters of FEFDFA with denoising and (optional) ReID loss. In all unsupervised settings, we use this training set (without ReID loss). It makes significant improvements. Specifically, in the experimental results between the second and first lines of Table 1, it can be observed that the performance of the unsupervised FEFDFA method has significantly improved compared to the baseline.  
• (2) "the parameters of the FEFDFA and baseline were trained alternately." Alternately training is a TRAINING TRICK for the supervised setting. Specifically, we (a) freeze the original parameters, train FEFDFA with denoising and reid losses, then (b) merge parameters of FEFDFA into the original parameters, train latest original parameters with ReID loss only, (c) repeat (a) and (b). In all supervised settings, we use this trick. It carries more improvement. Specifically, the experimental results in the second and third lines of Table 1. Note that step (a) still carries improvements in the supervised setting.  
• (3) "In addition, unless the baseline and FEFDFA are trained together, Eq. (12) is incorrect." Eq.(12) is CORRECT even if parameters of FEFDFA are trained and the original parameters are frozen. Both ReID loss and denoising losses can be used to optimize parameters of FEFDFA. This equals to (2)-(a) above. Similar ideas of "using task-related loss to supervise denoising module" can be found in DiffDet[4] and DiffSeg[5].

[1] Mukhopadhyay, Soumik, et al. "Diffusion models beat gans on image classification." arXiv preprint arXiv:2307.08702 (2023).

[2] Li, Alexander C., et al. "Your diffusion model is secretly a zero-shot classifier." ICCV 2023.

[3] Baranchuk, Dmitry, et al. "Label-Efficient Semantic Segmentation with Diffusion Models." ICLR 2022.

[4] Chen, Shoufa, et al. "Diffusiondet: Diffusion model for object detection." Proceedings of the IEEE/CVF international conference on computer vision. 2023.

[5] Tian, Junjiao, et al. "Diffuse Attend and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusion." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Q1: The authors missed an opportunity to benchmark their method against the latest advancements like CLIP-ReID.

A1: Thanks for your valuable feedback. CLIP is a very strong vision-text encoder, which is trained with 400 million data. Beyond CLIP, CLIP-ReID pioneeringly adapts CLIP, a zero-shot classifier, to ReID, a fine-grained image retrieval task. Our proposed method is model-free, thus it can be easily applied to CLIP-ReID without any modification. The experimental results of CLIP-ReID show stable improvement on the three datasets(DukeMTMC, MSMT, Market-1501). The code will be released if the manuscript could be accepted. We will add the experimental results to the revised manuscript. The experimental results are as follows, with mAP as the evaluation metric:

Q2: When contrasted with TransReID, this approach shows a moderate enhancement in performance.

A2: Thanks for your kind comment. Our proposed method is test-time computation-free (FREE LAUNCH!) and scalable to various baseline and downstream tasks, which is proven to be effective in CNN series (e.g. ResNet50), Transformer series (e.g. ViT, Vmanba), Person Re-ID (MSMT, Duke), Vehicle Re-ID (vehicleID), and Image Classification (ImageNet, CUB). For its scalability, we DO NOT hack it for backbones or tasks. With the same hyper-parameters, ours achieves consistent and stable improvement on ALL backbones and tasks above. For example, based on previous state-of-the-art CLIP-ReID, our proposed method achieves extra improvement from 0.8%-0.6% on Duke, MSMT and Market.

Q3: Is it necessary to use diffusion models for conducting ReID tasks, and what advantages do they offer compared to CNNs or ViTs?

A3: Thanks for your kind comment. In this work, we use the ViT series for conducting ReID tasks. Our proposed Feature Extraction and Feature Denoising Fusion Algorithm (FEFDFA) fuse the idea of diffusion into the backbone (e.g. ResNet, ViT). We treat each block of the backbone as a denoising layer and utilize the denoising ability of the diffusion model to denoise on a different feature extraction level. During the inference phase, the denoising layer parameters are fused with the backbone network parameters without incurring additional inference time cost.