Optimal Transport-based Labor-free Text Prompt Modeling for Sketch Re-identification

W1: Computational complexity.

A: Please refer to [Respones to nGMK:W1, edWJ:W6, beT2:W1] in Common Response.

W2, W3, W4: Minor issues.

A: Thank you for your thorough review and for highlighting these minor issues. We have revised the manuscript to address the mentioned issues.

Q1: Replaceability of VQA model.

A: Please refer to [Response to edWJ:W7, beT2:Q1] in Common Response.

Q2: $\boldsymbol{P}^\ast$.

A: Thank you for bringing up this question. Indeed, $P^\ast$ represents a weight matrix. Here is the corrected formula:

\begin{array}{c} \mathcal{L}_{tal}(R_i, S_i) = [m - D(R_i, S_i) + D(R_i, \hat{S}_h)]_{+} + [m - D(R_i, S_i) + D(\hat{R}_h, S_i)]_{+}, \\ D(R_i, S_i) = \gamma E(R_i, S_i) + (1 - \gamma)(1 - \boldsymbol{P}^{\ast}_{i,i})E(R_i, S_i) \end{array}

where, $\boldsymbol{P}^{\ast}$ is the optimal transport matrix with the least amount of cost, and $\boldsymbol{P}^{\ast}\_{i,i}$ represents the assignment weight of $(R\_i, S\_i)$ obtained after balancing the overall distribution.

Q3: Details of text attribute acquisition.

A: Thank you for raising this question. The text attributes acquisition occurs during the data processing stage, and does not lead to complex model calls during training. We have enhanced clarity in our manuscript to prevent such ambiguities.

Q4: Experimental comparisons.

A: Thank you for your constructive feedback. Based on task relevance, our adopted methods are all image-based re-identification methods, categorized into visible-infrared and sketch-based methods. The former include DDAG [54], CM-NAS [55], CAJ [56], MMN [57], DART [58], DCLNet [59], DSCNet [60], and DEEN [61]. The latter comprises BDG [6] and UNIReID [7]. We have already given the detailed descriptions of these experimental methods in the manuscript.

W1, Q2: Minor issues.

A: Thanks for your nice comment. We have identified all errors and corrected them in the manuscript.

W2, W3, Q4: Optimal transport.

A: Thanks for your constructive comment. Due to space constraints, a detailed description of optimal transport (OT) and matrix $\boldsymbol{P^\ast}$ can be found in the supplementary materials. Essentially, the OT problem resembles a "supplier-demander" issue: $\boldsymbol{\alpha}$ denotes the quantity of goods each supplier can provide, $\boldsymbol{\beta}$ denotes the quantity of goods each demander requires, and the cost matrix $\boldsymbol{C}$ indicates the expense incurred when each supplier supplies one unit of goods to each demander. The objective is to find the assignment solution that minimizes costs, yielding the optimal transport matrix $\boldsymbol{P^\ast}$. We add the description about how to calculate the $\boldsymbol{P^\ast}$:

For an input positive pair $(R\_i, S\_i)$ in a mini-batch $x$, the feature representations obtained through model inference are denoted as $f(R\_i)$ and $f(S\_i)$. If the sets of all sample features from different modalities in $x$ are treated as two discrete distributions, their alignment can be considered an optimal transport problem. The cost matrix $\boldsymbol{\hat{C}}$ is derived from pairwise feature similarities: $\boldsymbol{\hat{C}}\_{i,j}=[f(R\_i)^{\top}f(S\_j)]\_+$. We aim to acquire the optimal transport matrix $\boldsymbol{P}^{\ast}$ with the least amount of cost, where $\boldsymbol{P}^{\ast}\_{i,j}$ represents the assignment weight of $(R\_i, S_j)$ obtained after balancing the overall distribution. TAL can be represented based on triplet loss as the weighted sum of the original distance and the optimal assignment distance, dynamically updated at a certain rate $\gamma$:

\begin{array}{c} \mathcal{L}_{tal}(R_i, S_i) = [m-D(R_i, S_i)+D(R_i, \hat{S}_h)]_{+} + [m-D(R_i, S_i)+D(\hat{R}_h, S_i)]_{+}, \\ D(R_i, S_i) = \gamma E(R_i, S_i) + (1-\gamma)(1-\boldsymbol{P}^{\ast}_{i,i})E(R_i, S_i) \end{array}

where $\hat{R}\_h=argmax\_{R\_j\neq{R\_i}}D(R\_j,S\_i)$ and $\hat{S}\_h=argmax\_{S\_j\neq{S\_i}}D(R\_i,S\_j)$ are the most similar negatives in $x$ for $(R\_i, S\_i)$, $[x]\_{+}=\max(x,0)$, and $E(R\_i, S\_i)=\Vert{f(R\_i)-f(S\_i)}\Vert\_2$ denotes the Euclidean distance between feature representations.

Q1: The relationship between Reference [26]/[27].

A: Both papers [26-27] utilize optimal transport to address inter-modal distribution alignment in re-ID tasks. Ling et al. [26] select an optimal transport strategy and assign high weights to pairs with smaller intra-identity variation. Zhang et al. [27] first attempt to find an optimal transport matrix to re-weight the distance of different local parts in re-ID. These two papers inspire us to apply optimal transport theory for exploring more discriminative feature representations and establishing more reasonable standards for measuring sample distances in re-ID.

Q3: Noise.

A: Thank you for pointing out the ambiguity in this sentence. Compared to fixed handcrafted prompts, our dynamic knowledge learning mechanism does not rely on expert knowledge. This strategy can mitigate additional noise caused by inaccurate sentence templates, since our prompt include adaptable learnable components constrained by various loss constraints. We have revised it in the manuscript: "This integration introduces a dynamic knowledge learning mechanism that reduces noise introduction compared to handcrafted prompts, while enhancing flexible interaction across modalities and boosting the transferability of text embeddings."

W1: Computational complexity.

A1: Please refer to [Response to nGMK:W1, edWJ:W6, beT2:W1] in Common Response.

W2: Overfitting.

A2: We sincerely appreciate your valuable comments regarding overfitting, as this is also a concern for us. Consistent with the approaches in [1] and [2], we validate our model on two publicly available benchmark datasets, PKU-Sketch and Market-Sketch-1K, to ensure fair comparisons. To mitigate overfitting, we apply various data augmentation techniques, including random cropping, rotation, and style augmentation. Furthermore, to validate the model's robustness and generalization, we conduct supplementary evaluation experiments on two large-scale datasets (SYSU-MM01 and RegDB) for visible-infrared person re-ID, as shown in Table 1 of PDF. The results demonstrate that our OLTM achieves comparable performance in the visible-infrared domain.

W3: Image quality.

A3: Thank you for your insightful comments.

1. It should be clarified that the VQA model is only applied to RGB images, not sketch.
2. In fact, image quality affects not only the attributes generated by VQA, but also the performance of all visual tasks. Therefore, in addition to several data augmentation operations (e.g., random rotation, flipping, and style augmentation), we have also implemented the following strategies: 1）In contrast to the conventional method of directly generating a complete descriptive sentence，we use a "divide-and-conquer" strategy. By employing multiple questions to capture specific details (e.g., hair and shirt color), we effectively reduce potential noise in the text prompts. 2) Fixed attribute tokens are dynamically embedded into learnable prompts. In addition, applying various loss functions guides the model to prioritize reliable and valuable information. 3) To further refine fine-grained text embeddings, a "bin" mechanism is added to categorize potentially ambiguous content during Dynamic Consensus Acquisition (as analyzed in Sec. 4.2).

Q1: The number of learnable text prompts.

A1: Thank you for your question. As analyzed in Section 4.2, we extract $k$ text attributes $\\{att_1, att_2,..., att_k \\}$ using CLIP tokenizer to derive text tokens $a_i=Tokenizer(att_i)$, where each $a_i$ contains $n$ fixed numbers of tokens. Subsequently, $a_i$ preserves valid tokens, and is fed into CLIP token encoder to obtain the fixed attribute tokens $a_i \in{\mathcal{R}^{m\times{c}}}$. Here each word can be represented by $m$ valid tokens of dimension $c$ [3]. Those fixed parts are uniformly embedded into $l$ learnable prompts of dimension $c$, and finally we obtain final text description $q$. In our implementation details, the value of $k$ is $9$, and we set $n=77$ based on CLIP's fixed configuration. Thus the number of learnable text prompt is $n-k\times{m}$.

Q2: The fine-grained recognition ability of VQA model.

A2: Thank you for your valuable comment. The visual question answering (VQA) model has the capability to describe fine-grained recognition information for the following reasons: 1) The VQA model generates detailed attributes about various aspects of the pedestrian target (e.g., hair, backpack, hat), rather than relying on complete descriptive sentences. 2) We provide a visualization comparison, as shown in Figure 4 of PDF. This comparison demonstrates that using text attributes can guide effectively the attention of model.

Q3: The performance comparison with the handcrafted prompts.

A3: Thank you for pointing out the incompleteness in our ablation study. Our method remains competitive compared to handcrafted prompts mechanism[4], as shown in Table 4 below. The performance improvements can be attributed to the following reasons: 1) We introduce a dynamic learnable prompt mechanism without requiring additional expert knowledge. This enhances the model's adaptability and robustness. 2) VQA is more flexible compared to handcrafted prompts. It can generate more detailed information (e.g., glasses, hats) than what is included in handcrafted prompts.

Table 4: Comparison of prompt setting methods. "Handcrafted" and "VQA" denote manually annotated and VQA generated text attributes, respectively. "Template" represents the sentence template defined by experts. "Prompt" denotes the learnable text prompts.

Q4: Background information.

A4: Thanks for your constructive advice. This motivates us to investigate the feasibility of incorporating background information into text descriptions. Accordingly, we conduct additional background evaluation on Market-Sketch-1k dataset based on our initial studies. Specifically, we formulate the question 'What is the background of this image?' to extract textual attributes about background. The extracted background details are illustrated in Figure 2 of PDF. However, the introduction of background information result in a decrease of 2.81 in Rank-1 and 1.62 in mAP. This decline can be attributed to the absence of corresponding background information in the sketches, which potentially interferes with the model's learning process.

[1]Pang L, et al. "Cross-domain adversarial feature learning for sketch re-identification" ACMMM 2018.

[2]Lin K, et al. "Beyond Domain Gap: Exploiting Subjectivity in Sketch-Based Person Retrieval" ACMMM 2023.

[3]Radford A, et al. "Learning transferable visual models from natural language supervision" ICML 2021.

[4]Lin Y, et al. "Improving person re-identification by attribute and identity learning. Pattern recognition 2019.

W1: Triplet Assignment Loss.

A1: Thank you for your inquiry. We have included visual analysis in Figure 1 of PDF, which illustrates the convergence curve and sample distances during training. Figure 1(a) shows that conventional triplet loss converges prematurely. In contrast, our proposed triplet assignment loss exhibits higher volatility, reducing the risk of suboptimal local minima. Additionally, Figure 1(b) shows that a specific sketch sample (red box in the top left image) may have similar Euclidean distances to multiple RGB samples. The triplet assignment loss comprehensively considers the distribution of all samples (red box in the lower right image), offering broader possibilities for selecting the most relevant ones.

W2: Reference.

A2: We appreciate the reviewer’s recommendation. Inspired by this valuable work, we have cited this reference into Sec. 2 of the manuscript: "... fine-grained interaction. Furthermore, Li et al. [1] first attempt to conduct in-depth research on zero-shot multimodal ReID through a large foundational model. In this paper, we ...". However, since the source code of this reference is not available and it mainly focuses on zero-shot learning, we cannot make comparsion with it in experiments.

W3: Weights sharing.

A3: Thank you for your question. As stated in Sec. 4.3, Transformer blocks for different modalities do not share weights in feature enhancement module. The cross-modal interaction component in front of this module shared weights, facilitating the extraction of common features between modalities. Consequently, this module enables each modality to autonomously refine its representation. This strategy ensures precise adjustments according to specific needs and enhances the overall performance.

W4: The availability of text attributes.

A4: Thank you for pointing out this comment. To verify the effectiveness of different text attributes, we have provided additional ablation experiments in Table 5 below. The results show a significant decrease in model performance after discarding several hard-distinguished attributes (e.g., color and gender) in sketches. As shown in Figure 3 of PDF, sketches convey gender-related information through factors like body shape, and the contrast between light and dark areas effectively highlights specific color details. The TPR module injects detailed information into modal interactions during training. This enables the model to focus on these nuances autonomously, even without TPR during inference.

Table 5: The experiment results on various text attributes. "Gender", "Shirt" and "Pants" indicate whether the attribute includes gender, shirt color and pants color.

W5: Multi-query.

A5: Thank you for your question regarding the experimental setup. Similar to [2], "multi-query" involves combining multiple sketches of the same ID during both training and inference. Our paper employs a straightforward fusion method by averaging the image features from multiple sketches. Table 6 below provides a comparative analysis of various fusion strategies. The results demonstrate that the basic and simple fusion method achieves the best experimental performance.

Table 6: Performance comparison of different multi-query experimental methods.

W6: Computational complexity.

A6: Please refer to [Response to nGMK:W1, edWJ:W6, beT2:W1] in Common Response.

W7: Replaceability of VQA model.

A7: Please refer to [Response to edWJ:W7, beT2:Q1] in Common Response.

Q1: Minor concerns.

A1: We apologize for this typing mistake. We have revised it in the manuscript and added the following note to the supplementary material: "This section provides a detailed explanation of the cross-style retrieval. Given the extensive Market-Sketch-1K dataset, where each person is sketched by six different artists, notable variations exist across these sketches. Consequently, we devise this experiment to assess our model's resilience to diverse artistic styles. Experimental setups involve sketches labeled as S1 to S6, each originating from different artists. Models are trained on sketches by specific artists and tested on sketches by others. And, "single query" denotes separate queries for sketches by different artists of the same individual, while "multi query" indicates queries combining multiple sketches of the same person."

[1]Li H, Ye M, Zhang M, et al. "All in One Framework for Multimodal Re-identification in the Wild" CVPR 2024.

[2]Lin K, Wang Z, Wang Z, et al. "Beyond Domain Gap: Exploiting Subjectivity in Sketch-Based Person Retrieval" ACMMM 2023.