Parameter-efficient Fine-tuning in Hyperspherical Space for Open-vocabulary Semantic Segmentation

We sincerely thank you for your valuable feedback. We will include all new results and clarifications in the revised version.

Weaknesses

W1: Formula 5 does not specify how to interact with the $\boldsymbol{R}$ matrix.

A1: The interaction is introduced in Section 4.3. According to formula 6, we first treat all the matrices $\boldsymbol{R}$ in $l^{th}$ layer as a 3-order tensor $\mathcal{T}_{l}$.

Then, according to formula 7, we treat all the tensors $\mathcal{T}_{l}$ in parameter space as a 4-order tensor $\mathcal{T}$. After that, following formula 16, the interaction with the $\boldsymbol{R}$ matrix is achieved in $\mathcal{T}_w$ via two reversible relation matrices, i.e., $\mathbf{S}_3$ and $\mathbf{S}_4$.

Finally, in line with the manuscript, the tensor $\mathcal{T}_w$ is added in $\mathcal{T}$ by the updating rule $\mathcal{T} = \mathcal{T} + \alpha\mathcal{T}_w$. To sum up, formula 5 can be rewritten at the end of Section 4 (Methodology) as follows:

We will clarify this in the revised version.

W2: ... current fine-tuning strategies are usually asymmetrical, but it does not provide enough evidence or references ... .

A2: Many previous works [a-d] in the field of open-vocabulary semantic segmentation propose various types of asymmetric fine-tuning frameworks, where CLIP's text encoder is simply frozen, and the image branch is fine-tuned. We will provide these references in the revised version.

W3: In-depth analysis of how the misalignment problem impacts segmentation performance.

A3: Current fine-tuning methods for open-vocabulary segmentation are usually asymmetrical, i.e., typically freezing CLIP's text encoder and fine-tuning its image encoder. This strategy inevitably causes a potential obstacle: misalignment. More specifically, the misalignment arises from different alignment granularities. The text encoder maintains image-to-text alignment, while the image encoder shifts from image-to-text to pixel-to-text alignment. Due to these different alignment goals, the optimization process is largely impeded, leading to sub-optimal performance. We also provide some visualizations in Fig. 1 and Fig. 2 in the global author response PDF. One can observe that fine-tuning without alignment tends to separate the entire object region into a series of discrete regions due to the coarse granularity in understanding semantics.

W4: Mention SAM (Segment Anything) and how the current work is still significant.

A4: We have cited SAM as a large-scale foundation model in Section 2.2 ([23] in the main manuscript). Although SAM is an influential foundation model for image segmentation, adopting it for open-vocabulary semantic segmentation is non-trivial. The reason is the masks provided by SAM is class-agnostic, with no semantics. Assigning semantics from open-vocabulary set to masks usually also faces the challenge of misalignment due to different granularities of multimodalities. Therefore, our work is still significant.

Questions

Q1:Generalizable enough for other dense prediction tasks?

A5: Yes, our method is generalizable for object detection. To show this, we validate our method on an open-vocabulary object detection task. Please see Table 2 in the global response.

Q2:Discussing and comparing [1] [2].?

A6: The objective of [1] and our method is the same: to preserve both modality-shared and modality-specific information, but the proposed strategies are different. [1] achieves this goal by improving contrastive learning with several regularizations. However, its efficacy depends on delicately designed objectives and might cause optimization conflicts among them. In contrast, we propose a parameter-efficient fine-tuning strategy to preserve modality-specific information, where modality-shared information is achieved by DCRC. The official code of [1] is not available, and unfortunately, we have not received a response after emailing the first author of [1] until now.

OFT [2] aims to adopt orthogonal constraints on both CLIP's image encoder and text encoder to strictly maintain the original semantic structures. In contrast, we first apply orthogonal constraints only to the text encoder. This is crucial for open-vocabulary semantic segmentation, as it can provide more flexibility in fine-tuning the image encoder, facilitating the transfer of CLIP's initial alignment from image-level to pixel-level. We then introduce DCRC to encourage interactions between the encoders of the two modalities, further mitigating the misalignment issue. We compare our method with OFT [2] and demonstrate better performance, as shown below. We will include the above discussion in the revised version.

Limitations

L1: No qualitative comparison with the baselines and existing methods ... . It will be good to see how the results improve with and without alignment.

A7: Thanks for your question. We have provided some qualitative comparisons between our method and the existing SOTA method, i.e., CAT-Seg, on different datasets, as shown in Fig. 2, 3 and 4 in the main manuscript. To further validate the effectiveness of alignment, we present additional visual comparisons. Please see Fig.1 and Fig.2 in the global author response PDF.

[a] LANGUAGE-DRIVEN SEMANTIC SEGMENTATION. ICLR22.[b] Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP. CVPR22.[c] Side Adapter Network for Open-Vocabulary Semantic Segmentation. CVPR23.[d] SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentation. CVPR24.

We sincerely thank you for your valuable feedback and hope our following clarifications and responses could clear your concerns.

Weaknesses

W1: The introduction lacks transitions from existing problems to the approach of this paper, such as introducing the advantages of Hyperspherical Space.

A1: Introducing hyperspherical space in our method has two advantages. First, the hyperspherical space helps capture a model's intrinsic semantic structure. Specifically, by adhering to the hyperspherical energy principle to updating CLIP's text encoder, we preserve its intrinsic semantic knowledge, thus reducing the risk of over-fitting and improving performance on unseen classes. Second, the hyperspherical space provides a symmetric and robust parameter space for adapting CLIP, allowing its two encoders to mitigate the misalignment between the two modalities. We will add this content in the revised version.

W2: Some formula descriptions can be optimized, for example, explaining the meaning of * in Formula 9.

A2: The "*" represents a tensor product. We will add it to the revised version. Thanks!

W3: Details about comparison methods are needed. In Table 1, the compared method SAN includes an additional backbone.

A3: Thanks for your suggestion. We will correct this by filling in "side adapter" in the "Additional Backbone" column for SAN in the revised version.

Questions

Q1: This work focuses more on cross-modal alignment and the issues related to hyperspherical space, rather than on custom designs for pixel-level predictions. It appears to be more of a generalizable cross-modal fine-tuning paradigm. Have the authors attempted to validate it on tasks beyond open-vocabulary semantic segmentation?

A4: Thank you for your suggestion. We further validate our method on other tasks and observe consistent improvements. Please refer to the global response.

Limitations

L1: The authors provide no analysis of the limitations and broader impact. The author can analyze the limitations of this fine-tuning strategy in the field of OVS.

A5: Thank you for your comment. Our main contribution lies in proposing a parameter-efficient fine-tuning strategy for open-vocabulary semantic segmentation. However, we have not taken memory efficiency into account yet. Given the rapid evolution of vision foundation models for OVS, it is important to pursue low-cost deployment of fine-tuning, which could be improved in future work.

Thanks for your instructive comments. We will include all new results and clarifications in the revised version.

Weaknesses

W1: Limited novelty.

A1: We would like to emphasize that the novelty of our proposed POF mainly lies in its task-oriented design. Most previous OVSS methods opt for an asymmetric fine-tuning framework, which may easily lead to misalignment between the two modalities, thus impeding optimization speed[a]. In contrast, POF provides a symmetrical PEFT framework that unlocks a small number of parameters in hyperspherical space for encoders of the two modalities, largely mitigating this issue. We also visualize the training accuracy curve in Fig.3 of the global author response PDF, further demonstrating the advantage of the symmetric fine-tuning solution.

W2: Difference between POF and OFT[1].

A2: OFT[1] adopts orthogonal constraints on both CLIP's image and text encoders to strictly maintain the original semantic structures. Differently, POF applies orthogonal constraints only to the text encoder. This is significant for OVSS, as POF provides more flexibility in fine-tuning the image encoder, facilitating the transfer of CLIP's initial alignment from text-to-image to text-to-pixel. Besides, we compare OFT with the variant of our method, which solely uses POF, demonstrating the obvious improvement of POF over OFT. See the table below.

W3: In Eq.5, which module's weights are used?

A3: The weights are adjusted in the attention layer in line with most PEFT methods, e.g., LoRA.

W4: Why adopting reversible transformations to update the 4-order tensor in Eq.7, and its role ...?

A4: Through derivation in the manuscript and suppl., we conclude that the tensor-product between a pair of $p$-order tensors ($p \geq 3$) can be effectively converted into the matrix-product of internal 2-order matrices via $p-2$ reversible transformations $S_i(\cdot)$, $i=3,\cdots,p$. This indicates that $S_i(\cdot)$ can capture the correlations among the different matrices. In practice, we use $S_i(\cdot)$, $i=3,4$, to update the 4-order tensor in Eq.7, as it can efficiently achieve communication across modalities ($n_3$) and layers ($n_4$) in a parameter space. Besides, to capture potential non-linear relations among matrices, we achieve the reversible transformations $S_3(\cdot)$ and $S_4(\cdot)$ using two $k$-layer DNNs.

W5: ... also work in other fields?

A5: Yes. Please see Table 1, 2 in the global response.

W6: If the block diagonal structure is not adopted, ... reduce ... parameters?

A6: No, the block diagonal structure is necessary. For notation simplicity, we set $d_v=d_e=d$ in the Sec.4. However, in practice, the dimensions of the tunable matrix $R_{vi} \in \mathbb{R}^{d_v \times d_v}$ and $R_{ei} \in \mathbb{R}^{d_e \times d_e}$ are often not equal, e.g., $d_v=768$ and $d_v=512$ for ViT-B/16 version of CLIP. Given that the dimension of each matrix in a higher-order tensor must be consistent, we use a block diagonal structure to align the matrix dimensions between the two modalities, and thus, it cannot be discarded.

W7: Does the param ... calculate the decoder? Applicable to various decoders?

A7: Following the protocol used in previous PEFT works[b,c], we do not calculate the parameters of the decoder for all the methods. We will indicate this in the revised version. To further evaluate the effectiveness of our method under different decoders, we replace our decoder with three classical decoders: a linear probe, a CNN-based decoder[d] and a transformer-based decoder[e]. Results show our method is effective with different decoders. See the table below.

W8: Different VFM ... still valid?

A8: We base our method on fine-tuning another famous VFM, i.e., SAM (segment anything model), for adapting it to various downstream tasks. Note that SAM only has an image encoder, thus we remove the cross-modal interactions in DCRC. Besides, we incorporate orthogonal constraints in the learnable matrices of the lower layers of the encoder, as they contain generalizable representations of segmentation that should be preserved[f]. We follow the experimental settings provided in[g]. Our method shows competitive performance compared with other PEFT methods. See the table below.

W9: Comparing with more PEFT methods.

A9: We provide comparisons with more PEFT methods. The results are shown below. Our method achieves the best performance over all datasets.

[a]Misalign, Contrast then Distill:Rethinking Misalignments in Language-Image Pretraining.ICCV23.[b]Bridging Vision and Language Encoders:Parameter-Efficient Tuning for Referring Image Segmentation.CVPR23.[c]Time-,Memory-and Parameter-Efficient Visual Adaptation. CVPR24.[d]Pyramid Scene Parsing Network.CVPR17.[e]Segmenter:Transformer for Semantic Segmentation.ICCV21.[f]MMA:Multi-Modal Adapter for Vision-Language Models.CVPR24.[g]Parameter Efficient Fine-tuning via Cross Block Orchestration for Segment Anything.CVPR24.

I would like to keep the original score after reading through the rebuttal.

We truly thank you for the insightful comments and suggestions. We hope our responses can address your concerns.

Questions

Q1: Training time efficiency.

A1: We compare the training time of our method with other representative PEFT methods based on ViT-B/16 backbone, which shows comparable time costs. All results are obtained on 4 NVIDIA RTX 3090 GPUs. -- see the table below.

Q2: The effectiveness of this method on other CLIP-based tasks would provide strong evidence to support its effectiveness and potential broad impact.

A2: Thank you for your suggestion. In response to your comment, we compare our method with LoRA on other CLIP-based tasks and consistently outperform it, demonstrating the generalization of our method. Please refer to Table 1 and Table 2 in the global response.

[a] Visual prompt tuning. ECCV22.