SHE: Streaming-media Hashing Retrieval

Thanks for your valuable comments. Our responses are listed below.

R-Weakness-1: To enhance readability and understanding of our proposed SHE, we provide a detailed explanation of some key losses, which can be found at `R-Weakness-3’ in Reviewer TSiu.

R-Weakness-2: When mining the knowledge library, if all class prototypes are considered simultaneously, a sample may be influenced by multiple prototypes, leading to an unstable optimization objective. Thus, we only consider the nearest prototype. Besides, to verify the effectiveness of the above behavior, we conduct experiments on the XMedia and XMediaNet datasets with 128 bits by comparing the consideration of only the nearest prototype versus considering all prototypes. Specifically, we replace the KLM loss with the KLM* loss when mining the knowledge library, and the KLM* loss can be formulated as:

$L_{klm^*} = -\frac{1}{n_m} \sum_{i=1}^{n_m} \log \left( \frac{1}{K} \sum_{k=1}^{K} s\left(x_m^i | k\right) \right).$

The results are shown as follows:

The results show that only considering the nearest prototype with the same category when mining the knowledge library can enjoy a clear optimization objective and a better performance.

R-Suggestion-1: We provide detailed explanations of key loss terms in `R-Weakness-3’ of Reviewer TSiu.

R-Suggestion-2: We investigate the latest studies and found that there are no other CMH works to handle cross-modal retrieval (CMR) in streaming-media scenarios (SMC). Here, we discuss some real-valued representation-based CMR methods, including SDML[1] and DRCL[2], which are specifically designed to accomplish CMR in SMC. 1) These methods are real-valued based representations, which involve high computational complexity and memory overhead, making it difficult to meet the fast retrieval demands of large-scale datasets. 2) They deploy either a randomly initialized common space or an identical transformation weight matrix to guide semantic alignment, which overlooks the discrepancies among modalities. Since the randomly initialized common space or learned transformation weight matrix may retain modality-specific features that are not very relevant to the new modality, directly applying it may lead to semantic discrepancies. By contrast, our SHE transfers the baseline knowledge library to the new modality, thereby obtaining more semantically consistent representations.

R-Question-1: The KLM module can mine a knowledge library to preserve the semantic information of streaming data, which can serve as a medium to maintain semantic consistency while reducing the computational burden of maintaining historical data. The KLT module can transfer semantic information extracted from new modalities to the benchmark knowledge library. This allows our SHE to process new media without retraining the entire historical media data, thereby reducing computational complexity while ensuring semantic consistency between streaming modal data.

R-Question-2: We provide the explanation of why only considering the nearest prototype when mining the knowledge library, and perform a comparison with other methods. Please see `R-Weakness-2’ for details.

R-Question-3: To improve the efficiency of handling streaming-media data, MARS leverages the shared label parsing module to achieve semantic alignment without interaction across modalities. However, the label parsing module may retain modality-specific features that are less relevant to the new modality, and directly using it may lead to semantic bias. By contrast, our SHE proposes a KLT module to transfer the baseline knowledge library to the new modality, thereby obtaining more semantically consistent representations. To further demonstrate the advantages of SHE, we integrate the idea of using a knowledge library as a medium to maintain semantic consistency into MARS, and found it beneficial to do so. For detailed experiments and analysis, see `R-Question-2’ in Reviewer UdWL.

Reference

[1] Hu P, Zhen L, Peng D, et al. Scalable deep multimodal learning for cross-modal retrieval[C]//Proceedings of the 42nd international ACM SIGIR conference on research and development in information retrieval. 2019: 635-644.

[2] Pu R, Qin Y, Peng D, et al. Deep Reversible Consistency Learning for Cross-modal Retrieval[J]. IEEE Transactions on Multimedia, 2025.

Thanks for your constructive review. Our responses are listed below.

R-Weakness-1: To evaluate the quality of knowledge mined from different modalities, we design a score, which can be found at `R-Questions’ in Reviewer cJMB.

R-Weakness-2: To make readers understand the work more clearly, we reorganize the writing, such as the design of losses. Specifically, we explain Knowledge Library Regularization (KLR) loss and Knowledge Library Transfer (KLT) loss in detail, which can be found at `R-Weakness-3’ in Reviewer TSiu.

R-Weakness-3: It is a constructive suggestion. To solve this, we first design a score to evaluate the quality of knowledge mined from different modalities, which can be found in `R-Questions’ in Reviewer cJMB. Then, due to the word limit, we provide a simple solution here. If we observe that the score of the current modality is too low, we can wait for subsequent modalities to arrive. Once a modality with sufficient quality is detected, it can be selected as the initial modality. According to your suggestion, we will explore how to alleviate negative effects caused by low-quality first modalities in the future.

R-Suggestion-1: We investigate the latest studies and found that there are no other CMH works to handle cross-modal retrieval (CMR) in streaming-media scenarios (SMC). Here, we discuss some real-valued representation-based CMR methods, including SDML[1] and DRCL[2], which are specifically designed to accomplish CMR in SMC. 1) These methods are real-valued based representations, which involve high computational complexity and memory overhead, making it difficult to meet the fast retrieval demands of large-scale datasets. 2) They deploy either a randomly initialized common space or an identical transformation weight matrix to guide semantic alignment, which overlooks the discrepancies among modalities. Since the randomly initialized common space or learned transformation weight matrix may retain modality-specific features that are not very relevant to the new modality, directly applying it may lead to semantic discrepancies. By contrast, our SHE transfers the baseline knowledge library to the new modality, thereby obtaining more semantically consistent representations.

R-Suggestion-2: We reorganize the writing so that readers can understand the work more clearly, such as the design of losses. The details can be found at `R-Weakness-3’ in Reviewer TSiu.

R-Question-1: We have designed a score to evaluate the quality of knowledge libraries. Please see `R-Questions’ in Reviewer cJMB.

R-Question-2: As you suggested, we conduct experiments on the XMedia and XMediaNet datasets and found that it is effective to incorporate the above idea into existing methods (i.e., MARS). Specifically, we replace the original label parsing module in MARS with the knowledge library to guide representation learning and add the Knowledge Library Transfer (KLT) loss to maintain semantic consistency. The new objective of MARS can be formulated as:

$L=L_{ori}+L_{klt},$

where $L_{ori}$ is the original objective of MARS and $L _{klt}$ is the KLT loss. The results are shown as follows:

The results can show that using knowledge libraries as a medium to maintain semantic consistency can be integrated into existing methods.

R-Question-3: In fact, the KLR loss is not our main innovation, so we only provide a simple comparative Analysis. As you suggested, we provide another regularization technique, i.e., orthogonal regularization (OR). Specifically, OR aims to impose the constraint that inter-class prototypes are mutually orthogonal, which can be formulated as:

$L_{or} = \frac{1}{CK} \sum_{c=1}^{C} \sum_{k=1}^{K}\sum_{c'=1,c'\ne c}^{C} \sum_{k'=1}^{K} \left | \Gamma (p^{c,k}_m,p^{c',k'}_m) \right |,$

where $\Gamma$ is the cosine similarity function. We replace Knowledge Library Regularization (KLR) loss with OR loss and conduct experiments on the XMedia and XMediaNet datasets with 128 bits, and the results are shown as follows:

The results show that OR is effective but inferior to the regularization technique used in our paper. In the future, we will further explore better regularization techniques to contribute to our proposed SHE.

Reference

[1] Scalable deep multimodal learning for cross-modal retrieval, SIGIR. 2019: 635-644.

[2] Deep Reversible Consistency Learning for Cross-modal Retrieval, TMM, 2025.

We appreciate your valuable feedback. Our responses are listed below.

R-Weaknesses: To make the experiment more rigorous, a detailed description of the training procedure for the compared methods is provided. Specifically, there are two cases in the training. 1) For the Wikipedia and NUS-WIDE datasets, all baselines directly conduct training without any preprocessing or modification, as the data in these datasets consists of image-text pairs. 2) For the XMedia and XMediaNet datasets, all baselines except for MARS adopt label-based repeat sampling to construct pseudo-instance pairs, and then conduct training on every two modalities, as the data in these datasets consists of independent instances (unpaired and inconsistent in number). MARS directly conducts training without any preprocessing or modification, as it can process each modality independently.

R-Suggestion-1: We carefully check our paper and ensure that similar mistakes are not made.

R-Suggestion-2: To enhance the understanding of the loss of Eq. 8 (i.e., Knowledge Library Transfer loss, KLT), we add some explanations to further elaborate on it. The KLT loss is formulated as:

$L_{klt} = -\frac{1}{CK} \sum_{c=1}^{C} \sum_{k=1}^{K} \log \left [ min(1,max(0,\Gamma(p_1^{c,k},p_m^{c,k}) - \sigma +1)) \right ].$

KLT aims to transfer the semantic information in the newly mined knowledge to the benchmark knowledge library, thereby achieving cross-modal semantic consistency. As is shown in the above equation, we encourage the prototypes mined from the newly coming modality to align with the prototypes with the same semantics in the benchmark knowledge library by maximizing their similarity. Notably, we focus on maintaining a certain level of semantic similarity between them by setting a similarity boundary rather than forcing them to be identical. That is, the above learning objective is $\Gamma(p_1^{c,k},p_m^{c,k}) \ge \sigma$, which relaxes the alignment requirement and can further improve the representation ability of multimodal data.

R-Questions: It is a constructive suggestion. To alleviate negative effects caused by low-quality first modalities, we first design a score to evaluate the quality of knowledge mined from different modalities. For the $m$-th modality, the score could be formulated as follows:

$S_{m}=\frac{1}{2n_m} \sum_{i=1}^{n_m} \max_{k=1, \ldots, K} s\left(x_m^i | k\right) +\frac{1}{2CK} \sum_{c=1}^{C} \sum_{k=1}^{K} \frac{ \sum_{k'=1,k' \ne k }^{K}e^ {\Gamma(p^{c,k}_m,p^{c, k'}_m)}} {\sum _{c'=1}^{C}\sum _{k'=1}^{C}{e^ {\Gamma(p^{c,k}_m,p^{c',k'}_m)}}-e},$

where $s(x_m^i | k)=\frac{\sum_{c=1}^{C} y_m^{i,c}\cdot \Gamma(b_m^i,p^{c,k}_m)+1}{2}$ and $\Gamma(\cdot,\cdot)$ is the cosine similarity function. In the above equation, the first item is responsible for reflecting the consistency between the mined knowledge and sample representations, while the second item is responsible for reflecting the distinction among inter-class knowledge. The higher the score is, the more the mined knowledge reflects the semantic information in the samples and the more discriminable it is, that is, the higher quality the modality owns. To verify the effectiveness of the proposed score, we conduct experiments on the XMedia dataset with 128 bits. The results are shown as follows:

From the results, it can be observed that the proposed score can accurately reflect the quality of the modalities. Due to the word limit, a simple strategy is provided here. In real applications, if we observe that the score of the current modality is too low, we can wait for subsequent modalities to arrive. Once a modality with sufficient quality is detected, it can be selected as the initial modality. According to your suggestion, we will explore how to alleviate negative effects caused by low-quality first modalities in the future.

Thanks for your valuable comments. Our responses are listed below.

R-Weakness-1: We have revised this issue, providing the relevant explanation when KLR first appears.

R-Weakness-2: In the introduction section, we list CMH methods (Such as UCCH, DHaPH, DCH-SCR[1], SCH, and CICH[2]) to illustrate the shortcomings of existing CMH research. Here, we briefly discuss some recent CMH works below: 1) UCCH aims to implicitly capture the semantic relevance between modalities by mining the co-occurrence relationship between multimodal data (such as image-text pairs) and their initial distribution characteristics. 2) DCH-SCR[1] deploys a ranking alignment mechanism to capture the close semantic relationship between modalities, which preserves the semantic similarity between tags and feature levels. 3) CICH[2] designs a prototypical semantic similarity coordination module to globally rebuild partially-observed cross-modal similarities under an asymmetric learning scheme. Notably, these CMH methods implicitly assume that all modalities are prepared before processing and adopt joint learning to achieve cross-modal semantic alignment. In practical application scenarios, it is challenging to collect data of all modalities simultaneously, such as emergency medical aid, multi-modal search engines, and financial data analysis. More commonly, data from all modalities is collected asynchronously. To improve the practical applicability of CMH, this paper proposes a novel method termed SHE to enhance the flexibility of processing asynchronously collected multimodal data.

R-Weakness-3: To enhance the readability and understanding of our SHE method, we explain some loss functions in detail. 1) For the Knowledge Library Regularization loss (KLR), which is formulated as:

$L_{klr} = -\frac{1}{CK} \sum_{c=1}^{C} \sum_{k=1}^{K} \log \frac{ \sum_{k'=1,k' \ne k }^{K}e^ {\Gamma(p^{c,k}_m,p^{c, k'}_m)}} {\sum _{c'=1}^{C}\sum _{k'=1}^{C}{e^ {\Gamma(p^{c,k}_m,p^{c',k'}_m)}}-e}.$

KLR aims to escape a simple solution where all prototypes converge to a single point. As the above equation shows, we first treat inter-class prototypes as positive pairs and intra-class prototypes as negative pairs. Then, we maximize the similarity among positive pairs and minimize the similarity among negative pairs, thereby enhancing the distinctiveness between inter-class prototypes while ensuring semantic consistency between intra-class prototypes. 2) For the Knowledge Library Transfer loss (KLT), which is formulated as:

$L_{klt} = -\frac{1}{CK} \sum_{c=1}^{C} \sum_{k=1}^{K} \log \left [ min(1,max(0,\Gamma(p_1^{c,k},p_m^{c,k}) - \sigma +1)) \right ].$

KLT aims to transfer the semantic information in the newly mined knowledge to the benchmark knowledge library, thereby achieving cross-modal semantic consistency. As is shown in the above equation, we encourage the prototypes mined from the newly coming modality to align with the prototypes with the same semantics in the benchmark knowledge library by maximizing their similarity. Notably, we focus on maintaining a certain level of semantic similarity between them by setting a similarity boundary rather than forcing them to be identical. That is, the above learning objective is $\Gamma(p_1^{c,k},p_m^{c,k}) \ge \sigma$, which relaxes the alignment requirement and can further improve the representation ability of multimodal data.

R-Weakness-4: Parallel training means that the proposed SHE can independently process each modality. When facing datasets with five modalities, SHE can directly perform training without any preprocessing, resulting in $5$ training processes and a storage overhead of $5$ sub-networks. By contrast, all baselines except MARS are limited to joint training on paired data from two modalities. When these methods face datasets with five modalities, we first construct pseudo-instance pairs through label-based repeat sampling and then conduct training on every two modalities. As a result, they require $5\times 4/2 = 10$ training processes and incur a storage overhead of $5 \times 4 = 20$ sub-networks. Obviously, parallel training can enhance training efficiency and reduce the required storage space. Besides, to further show the advantages of parallel training, we compare the training time (TT) and storage overhead (SO) of SHE and SCH on the XMedia dataset as follows:

The results show that our SHE can enhance training efficiency and reduce the required storage space by parallel training.

R-Suggestions: We carefully check our paper and ensure that similar mistakes are not made.

Reference

[1] Liu X, Zeng H, Shi Y, et al. Deep cross-modal hashing based on semantic consistent ranking[J]. IEEE Transactions on Multimedia, 2023, 25: 9530-9542.

[2] Luo H, Zhang Z, Nie L. Contrastive incomplete cross-modal hashing[J]. IEEE Transactions on Knowledge and Data Engineering, 2024, 36: 5823-5834.