Prompt Learning with Quaternion Networks

We appreciate your valuable feedback and address your concerns as follows.

W: Model structure and performance. As suggested, we will update Figure 2 to supply more detailed comparisons to help readers understand the differences between our method and other approaches. In general, both MaPLE and our proposed QNet are based on the architecture of CLIP. MaPLE primarily employs linear projections to align and merge features from different modalities. Linear projection can maintain mutual synergy and reduce the tendency to learn independent uni-modal prompts, proving effective in specific situations. However, it oversimplifies the intricate interconnections in multimodal data. In our approach, QNet handles diverse modalities by leveraging the inherent orthogonal relationship among the three imaginary axes within the quaternion hidden space. Unlike linear projection in MaPLE, our QNet utilizes the quaternion space to construct the intricate and non-linear associations between modalities, which yields better performance in multiple datasets, especially in zero-shot learning scenarios.

We appreciate your valuable comments and address your concerns as follows.

Cons1: Computation overhead and latency. For a fair comparison, we calculated the parameters and GFLOPs for both MaPLe and our proposed QNet, which use multimodal prompts. Our method involves fewer trainable parameters (2.93M) than MaPLe (3.56M) yet achieves better performance across 11 datasets. Regarding computational overhead, while MaPLe operates at 167 GFLOPs, our method runs a slightly higher computational load at 179 GFLOPs. This slight increase is attributed to the more complex non-linear quaternion space compared to the linear space of MaPLe. As for latency, CoOP, Co-CoOp, Maple, and our QNet runs at 299Fps, 48Fps, 103Fps, and 50Fps under processing a hundred images per batch. Among these methods, MaPLe and CoOp exhibit higher speeds as they only require a single forward pass of prompts through the text encoder. In contrast, Co-CoOp and our QNet, which rely on instance-conditional designs, demand an independent forward pass for instance-specific prompts. Hence, Co-CoOp and our QNet consume more GPU memory when the batch size is larger than 1, leading to slightly slower running speeds.

Cons2: The superiority of quaternion spaces. We acknowledge that quaternion hidden spaces are more sophisticated than linear spaces, resulting in better performance. However, current multimodal fusion methods often rely on explicit interaction structures that struggle to fully capture the varied and complex patterns inherent in multimodal data. For example, MaPLE with linear mappings oversimplifies the associations between different modalities. Similarly, the cross-attention method assigns maximum weight to the modality that is beneficial to the current task, but fails to leverage the rich complementary information among different modalities, inadvertently discarding essential data. We have presented detailed ablation experiments in Appendix Figure B. In our approach, QNet, we use quaternion hidden space to construct more complex and non-linear relationships among diverse modalities in a more compact manner. First, we merge output image features from a pre-trained model with text context features. Then, we process these integrated features through a quaternion encoder. Within the quaternion hidden spaces, our QNet employs orthogonal imaginary axes (i.e., i, j, and k) to further tackle the fused features in a higher dimension. Our approach provides a holistic understanding and captures intricate details that might be overlooked from a single-dimensional perspective. Additionally, our QNet integrates features across hierarchical levels to enrich the representation from various perspectives. Extensive experiments on multiple datasets demonstrate the superiority of our approach over the state-of-the-art methods. Furthermore, our QNet can be used as a plug-and-play module to support various multimodal approaches.

We appreciate your valuable feedback and address your concerns as follows.

W: Expanded comparisons and evaluations. Our method, QNet, significantly differs from directly combining Quaternion networks and the existing prompt learning approach, where all real and imagery axes are used as parameter axes. Instead, our QNet only uses imagery axes in which the i-axis and j-axis act as parameter axes, and the k-axis acts as a modulation axis to balance the weights of the i-axis and j-axis. Experiments in Table 5 and Table 6 demonstrate the effects of different real and imagery axes, affirming that using imagery axes can better learn multimodal information from the fused features. We also compared our method with Coop and MaPle, both with the setting of a single layer in Table 4, which further validates the effectiveness of modality fusion in QNet. As suggested, we extend our QNet to related tasks, such as video understanding. Due to limited time, we conducted preliminary experiments on the UCF-101 video dataset (see Table 1). Using the ViFi-CLIP's base-to-novel generalization setting, QNet was applied to a Kinetics-400 pre-trained ViFi-CLIP with prompt learning. QNet outperforms the IVLP method and even exceeds fully fine-tuned models like ActionCLIP, indicating its strong generalization ability across various modalities, including video tasks. Moreover, we will make more detailed comparisons with other methods and conduct more evaluations on other tasks to validate the effectiveness of QNet in the revised version. We gratefully appreciate your suggestions to improve this work.

Table 1: Performance comparison in video action recognition generalization benchmark on UCF-101. We employ QNet and IVLP on ViFi-CLIP and compare with the prior video approaches.