PLIP: Language-Image Pre-training for Person Representation Learning

Thanks for your valuable comments and appreciation of our work. The concerns are answered as follows.

(i) Pre-training on commonly used ViT.

Due to the need for handling multi-scale image features in the TIC task of PLIP, we only consider the variants of vision transformers with hierarchical structure design, such as Swin Transformer, as the backbone in our PLIP. To verify the effectiveness, we perform a range of experiments with Swin Transformer as the backbone and achieve a series of results, shown in Table 1, Table 5, Table 7, Table 9, Table 18 and Table 21 of our paper. These experimental results all demonstrate the effectiveness of our pre-training method on the Swin-Transformer. Certainly, in our future work, we will explore more universal pre-training methods that impose fewer constraints on the model architecture. We hope the above response has addressed your concerns.

(ii) More concise presentation.

Thanks for pointing this out. Due to the extensive experiments conducted to validate the effectiveness of each component in the proposed framework, as well as experiments on a wide range of downstream tasks, the length of this paper is considerable. Following your suggestion, we will refine our presentation with greater conciseness in the future version.

(iii) The reason for inconsistent improvements.

Thanks for your insightful comment. Due to LGUR's high performance, other pre-trained models can achieve the best results on this method, yet our PLIP obtains the best results on CMPM/C, the reasons for which are explained below:

A series of previous works in the field of multi-modal pre-training have demonstrated that, for pre-trained multi-modal models, excellent performance can be achieved with simple fine-tuning on downstream tasks, and additional complex model structure design is unnecessary. We must note that CMPM/C is a very simple algorithm that correlates the visual and textual global representation by designing only two loss functions, without any additional complex design for model structure. On the other hand, SSAN and LGUR are designed with highly complex additional structures for model training. In fact, through large-scale person-centric multi-modal pre-training, PLIP has learned a very discriminative multi-modal shared space for text-based person Re-ID. Therefore, simply fine-tuning PLIP with CMPM/C gives better results than SSAN and LGUR.

(iv) Fine-grained image with a vague text.

Thanks for your valuable comment. In fact, due to the lack of sufficiently fine-grained texts in existing manually annotated datasets, there will inevitably be some relative vague texts in our synthetic dataset SYNTH-PEDES. We acknowledge that relatively vague texts naturally lead to the model not being good at distinguishing some very detailed features to a certain extent. However, we must note that our model has a preliminary understanding of the meaning of attributes and colors, and can associate them with related image regions. This ability to distinguish between different parts of the person body leads to discriminative person representation learning to some extent. We believe that not being good at distinguishing very detailed features will not have a significant negative impact on person representation learning. Obviously, if we want to further enhance the model's ability to perceive details, we can try manually annotating datasets with super fine-grained texts. We will continue to work towards this research direction with diligence.

Thanks for your valuable comments and appreciation of our work. The concerns are answered as follows.

(i) Somewhat complex designs.

Considering that existing general-domain multimodal pre-training techniques do not take into account person-related characteristics, they are not well-suited for application in person representation learning. Therefore, we propose a novel language-image pre-training framework termed PLIP, with three elaborately designed three pretext tasks. These three tasks are obviously effective without introducing significant computational overhead.

(a) Effectiveness. The effectiveness of these three tasks has been demonstrated across a series of experiments in this paper, with each task significantly enhancing the performance of the learned representations. For instance, as shown in Table 18, under a fair comparison setup with ResNet50 as the backbone, our PLIP outperforms CLIP by 14.8% and 11.0% in Rank-1 accuracy on the CUHK-PEDES and Market1501 datasets, respectively.

(b) Computational overhead. Under the same experimental conditions, training the ResNet50 (or ViT-Small) for 70 epochs on the entire SYNTH-PEDES dataset using 4×Geforce 3090 GPUs takes approximately 12.4 days for CLIP, 18.1 days for BLIP, and 15.2 days for PLIP. It can be seen that our PLIP achieves significant performance improvement without introducing much additional training cost. Notably, when directly transferring to the CUHK-PEDES dataset, the rank-1 results of BLIP and PLIP are 44.9% and 52.9%, respectively, while the training time of PLIP is 2.9 days less compared to BLIP.

Certainly, exploring how to simultaneously improve the efficiency and performance of pre-training methods is a highly meaningful research topic. Following your suggestion, we will continue to advance this in our future work.

(ii) Noisy data.

Due to the impracticality of manually annotating all images, noisy data is a common issue encountered by almost all large-scale image-text datasets. To minimize the amount of noisy data in the dataset as much as possible, we employed a series of denoising strategies to ensure the quality of the dataset to the greatest extent. The specific details of the denoising strategies can be found in Section A.7 of the paper. Meanwhile, we meticulously evaluated the quality of the dataset in Section 9.2 of the paper. The evaluation results indicate that the quality of our dataset is nearly on par with that of manually annotated datasets.

(iii) Typo.

Thanks for pointing this out. Following your suggestion, we will carefully review the full text to avoid these typo errors.

(iv) About the prediction head of IAP.

Thanks for your insightful question. The IAP task requires to predict the masked words in a textual description. Considering the entire vocabulary as the sum of categories, with each word as a separate category, the IAP task is akin to, given a masked position, selecting the most fitting word from the entire vocabulary. In essence, this is a multi-class classification task, and therefore we use the common multi-class classification head instead of multiple binary classification heads to predict the words.

(v) About TIC task.

The pretext task of text-guided image colorization is first proposed in this paper and has been proven to be very effective for person representaion learning. It forces the model to understand the meaning of colors and attributes, and associates them with related visual regions, rather than simple memorization. This ability to distinguish between different parts of the person body leads to more discriminative person representation learning and guarantee the superior performance on many person-centric tasks.

Thanks for your valuable comments. The concerns are answered as follows.

(i) Core difference between PLIP and Lapscore.

In fact, there are many differences between our PLIP and Lapscore.

(a) The different core motivation. The core motivation of Lapscore is to design some modules specifically for the single task of text-based person Re-ID. Unlike Lapscore, our PLIP is not limited to a single task; instead, it is designed to learn general person representations capable of boosting various person-related tasks.

(b) The different core designs. Lapscore designs two color reasoning modules for text-based person Re-ID. The first uses LSTM to obtain the text embedding, which is then infused into each skip connection of the UNet architecture to achieve color restoration of grayscale images. The second utilizes MobileNet to extract features from color images, and then employs these features along with the Bilinear Attention Network to predict the masked color words in the text. These complex designs are challenging to apply directly to large-scale pre-training. However, our PLIP not only deeply considers the characteristics of persons, but especially make targeted, simple and efficient representation learning designs for each task, making it naturally adaptable to large-scale pre-training.

(ii) Some unclear representations.

Thanks for pointing this out. The unexplained abbreviations in Figure 2 are an oversight. Specific explanations can be found in the second response to Reviewer Rk98. Notably, the methods as baselines in Table 1 are actually described detailedly in the table caption.

(iii) Writing.

This paragraph dicusses the challenge of aligning fine-grained text with image regions, a future research direction for us. We will simplify the expression to make it more easily understandable. Also, we will polish the writing to avoid any issues such as repetitive images.

(iv) Experimental evaluation.

We respectfully disagree with that the experimental evaluation is incomplete, and certain comparisons lack logicality and fairness. In fact, we have already conducted verification experiments for both the method and dataset.

(a) We conduct comparative experiments on the pre-training method. In Table 18, to verify the superiority of our PLIP, we have pre-trained a series of models on the entire SYNTH-PEDES dataset with different pre-training methods, and compare their transfer performance on downstream datasets. The results show that our PLIP outperforms all other pre-training methods significantly on CUHK-PEDES and Market1501. Considering that the pre-training datasets are identical, these experimental results strongly verify the effectiveness of our method.

(b) We perform extensive ablation studies to verify the effectiveness of the pre-training dataset. In Section A.9.3, through three experiments, we meticulously explore the effectiveness of various components of our dataset. A series of experimental results all indicate that pre-training on our dataset significantly enhances the learning of more generalized person representations. Considering that the pre-training methods are identitcal, these experimental results strongly verify the effectiveness of our dataset.

Additionally, thanks for pointing out these related works. Our work achieves comparable or leading performance compared to them, and we will include the comparative results in our future version.

(v) Advantages of specialized person image captioner and a discussion on existing methods.

It is an interesting question. We believe that using a specialized person image captioner to annotate person images has the following clear advantages over using existing MLLMs.

(a) The generated texts have higher quality. MLLMs are trained on extensive general-domain image-text datasets, which include a limited number of person images. The texts generated by MLLMs are often coarse, with low granularity, and are more likely to contain erroneous descriptions, which has been discussed in [4]. These low-quality texts are disadvantageous for model pre-training. However, by training on person-related image-text datasets, our image captioner is able to provide significantly more accurate descriptions of the appearance of persons. These accurate textual descriptions are a guarantee of our dataset's high quality.

(b) The speed of text generation is significantly faster. Existing MLLMs generally have a large number of model parameters and slow inference speeds, which can lead to considerable time costs when generating text for large-scale images. In contrast, our captioner is a small expert model that generates text quickly and is well-suited for generating text for large-scale images.

Additionally, there is a discussion of the image captioning methods.

[1] annotates images by having MLLMs fill in attribute words into human-designed templates. [2] uses CLIP to calculate the similarity between images and various attribute prompts to obtain person attribute information, which is then filled into a given template to produce final textual description. [3] first queries MLLMs for each attribute to obtain attribute information, and then either fills this information into a given template or further processes it with a language model to generate final textual description.

We can observe that these methods essentially leverage existing models to tag images with attribute labels. Since the utilized models, such as CLIP, have not been trained on specialized person data, there is a common issue of less accurate attribute tagging. In contrast, our captioning method is capable of generating more accurate and stylish textual descriptions. We will include a more detailed discussion in the paper.

(vi) Blind review rules.

After review, it has been found that this sentence does not violate the rules.

Reference

[4] 2024-CVPR-UFineBench: Towards Text-based Person Retrieval with Ultra-fine Granularity

Dear Reviewer RKFb:

We thank you for the precious review time and valuable comments. We have provided corresponding responses, which we believe have covered your concerns.

The discussion phase has been on for several days and we have not heard any post-rebuttal responses from you yet.

We would love to convince you of the merits of our paper. We hope to further discuss with you whether or not your concerns have been addressed. Please let us know if you still have any unclear parts of our work.

All the best,

Authors.

Thanks for your valuable comments. The concerns are answered as follows.

(i) Analysis of experimental comparison fairness.

We believe that it might be due to some of our wording that led to your misunderstanding that our method would result in information leakage and issues of unfair comparison. In fact, our method does not have an issue with information leakage, and a series of fair experimental comparisons have also demonstrated the superiority of our approach. The detailed explanations are as follows.

(a) We utilize the training sets of the CUHK-PEDES and ICFG-PEDES dataset to train our SPAC, ensuring that there is no information leakage from the test sets of these two datasets.

Specifically, in our method, the training sets from these two datasets are used to train our SPAC. Then, we employ SPAC to generate corresponding text descriptions for large-scale person images. Subsequently, we use these synthetic image-text pairs to train our PLIP model. Throughout this process, we do not utilize any information from the test sets of CUHK-PEDES and ICFG-PEDES. Therefore, when we validate the excellent performance of our method on the test sets of these two datasets, there is no issue of information leakage.

(b) A series of experimental results demonstrate the superiority of our method in promoting various person-related tasks on many other datasets.

For example, on the unsupervised person ReID task, using PPLR as the baseline algorithm, our PLIP pre-trained model significantly boosts the mAP by 5.1% on Market1501 and 14.7% on MSMT17. It is worth noting that the Market1501 and MSMT17 datasets are not encountered during our pre-training process, hence there is no issue of information leakage. This further demonstrates that our PLIP can learn highly generalized person representations.

In conclusion, our pre-training framework avoids information leakage, and various fair comparative experiments have also confirmed its superiority. Meanwhile, thanks for pointing out the possible misinterpretations. We will clarify our use of the combined CUHK-PEDES and ICFG-PEDES training sets at line 1046.

(ii) Undefined abbreviations.

Thank you for pointing this out. The undefined abbreviations in Figure 2 are an oversight. In fact, SIC should be TIC (Text-guided Image Colorization), VLC should be IVLC (Identity-based Vision-Language Contrast), and VAP should be IAP (Image-guided Attributes Prediction). We'll fix these and clarify them in the caption.

(iii) Comparison with latest works.

Firstly, the methods in Tables 6 and 7, though from 2022, are top pre-training models in person ReID. Secondly, to reproduce these methods for our experiments, they must have open-sourced their models and code, as we assess the impact of various pre-trained models on downstream unsupervised tasks, like in Table 6. Therefore, we compare key methods such as LUP, LUPNL, and PASS in these tables.

Indeed, in many of the experiments, we have compared our PLIP with the latest works, such as SOLIDER (CVPR 2023) in Table 1, and IRRA (CVPR 2023), APTM (ACMMM 2023), RASA (IJCAI 2023) in Table 5. Notably, compared to the SoTA expert models for these different tasks, our PLIP has achieved consistently competitive or leading performance.

(iv) Novelty.

We respectfully disagree with that the tasks of our PLIP are lack of novelty and unlikely to provide valuable new insights for subsequent research in this field.

It is worth noting that the three pretext tasks we proposed are not simply equivalent to the three works mentioned. In fact, there are significant differences between them.

(a) The core motivation significantly differs from these works. The core motivation of all these works mentioned is to design various expert models specifically for the single task of text-based person Re-ID. However, our motivation is to propose a language-image pre-training framework to advance a series of person-related tasks rather than a single task. Alternatively put, our PLIP, unlike these works, is not confined to one specific task but is proposed to learn general person representations that can boost a wide range of person-related tasks.

(b) The specific designs and implementation differ from these works. These works have comparabaly complex structures and loss function designs to maximize the performance on the single text-based person Re-ID task. These complex specialized designs are quite time-consuming and challenging to apply directly to large-scale pre-training. However, our PLIP not only deeply considers the characteristics of persons, but especially designs targeted, simple and efficient representation learning tasks, making it naturally adaptable to large-scale pre-training.

Furthermore, our PLIP opens up a new research direction on how to utilize language information to learn general person representations suitable for various person-related tasks. We believe that our work can provide valuable insights for subsequent research from many aspects. For example:

(a) How to generate more detailed fine-grained textual descriptions for images?

(b) How to design more effective multi-modal pre-training tasks for person representation learning?

(c) How can we add more modalities to pre-training? Does this improve model performance?

(d) How to optimize pre-training data: balancing quantity and quality, and maximizing implicit knowledge extraction?

These are the directions our team will pursue. I believe researchers in our field will build on our work to explore and answer these questions.

(v) Masking strategies.

In fact, in the IAP task, we randomly mask both attribute phrases and other non-attribute phrases, with attribute phrases being masked at a higher probability. We believe this approach encourages the model to focus more on extracting attribute information from person images. Meanwhile, we have conducted ablation experiments on different masking strategies in Section A.9.4 of our paper.

Dear Reviewer Rk98:

We thank you for the precious review time and valuable comments. We have provided corresponding responses, which we believe have covered your concerns.

The discussion phase has been on for several days and we have not heard any post-rebuttal responses from you yet.

We would love to convince you of the merits of our paper. We hope to further discuss with you whether or not your concerns have been addressed. Please let us know if you still have any unclear parts of our work.

All the best,

Authors.