Enhancing Robustness of Vision-Language Models through Orthogonality Learning and Self-Regularization

1. The method lacks originality, as regularization strategies similar to this have been previously explored in papers such as KgCoOp [1] and PromptSRC [2].
2. The empirical performance gains are minimal; compared to PromptSRC, this method only achieves a 0.05 improvement.
3. Lacks of comparisons with full fine-tuning methods, CLIP-CITE [3], CLIP-ood [4].

[1] Self-regulating prompts: Foundational model adaptation without forgetting. ICCV 2023 (This also is incorrectly cited as CVPR 2023 in the manuscript.)

[2] Visual-Language Prompt Tuning with Knowledge-guided Context Optimization. CVPR, 2023.

[3] Fully Fine-tuned CLIP Models are Efficient Few-Shot Learners. arXiv, 2024.

[4] CLIPood: Generalizing CLIP to Out-of-Distributions. ICML, 2023.

In the methodology section, the main idea lacks novelty. By comparing Eq. 2-3 in this paper with Eq. 1-2 in paper [1], they are almost the same. It appears to be only a naive extension of the orthogonal fine-tuning idea from text-to-image model generation to image recognition. What unique advantage does the orthogonal fine-tuning method offer compared to existing prompt-tuning methods?

The self-regularization basically constrains the fine-tuned model weights from deviating from the pretrained model weights, which has already been investigated in paper [2] and paper [3]. Additionally, the CutOut augmentation is a straightforward technique to enhance the diversity of image inputs, with similar ideas appearing in paper [4].

The improvement in experimental results is quite marginal. From Table 1, we see that, compared to the current baseline method, PromptSRC, the base category accuracy of the proposed method actually drops, and the new category accuracy (76.55 vs. 76.10) and HM category accuracy (80.02 vs. 79.97) show less than a 1% improvement. The performance gains marked in the table are misleading, as the comparison with CoOp is not fair. Papers [5] and [6] represent the state-of-the-art in prompt tuning, yet the author did not include them as baselines. If these two papers were included, the author’s results would be even less competitive.

Furthermore, from Table 5, the efficiency comparison shows that the proposed OrthSR method has significantly more parameters than CoOp but the same FPS. This table does not show any advantage of the OrthSR method. Additionally, Table 5 lacks an efficiency comparison with PromptSRC.

[1]Controlling Text-to-Image Diffusion by Orthogonal Finetuning. [2] CLIPood: Generalizing CLIP to Out-of-Distributions [3]Self-regulating Prompts: Foundational Model Adaptation without Forgetting [4] Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models [5] Dept: Decoupled prompt tuning, CVPR2024 [6] ArGue: Attribute-Guided Prompt Tuning for Vision-Language Models, CVPR2024

1. The experimental performance is not competitive compared to recently published methods.
2. The motivation for using orthogonal fine-tuning is not clear.

1. Unclear Motivation:

While the paper argues that its aim is to enhance the robustness of models like CLIP, it is never clearly demonstrated how robustness is measured in the scope of the discussion, and why OrthSR is capable of improving over previous methods. Adding orthogonal matrices is an interesting attempt, but there is limited discussion on its similarities and differences compared to LoRA, a main target of comparison in the paper that also tries to fine-tune the pre-trained weights. Moreover, the section on cutout augmentation seems to be somewhat unrelated to the formulation of the proposed approach.

2. Limited Improvements:

While the proposed approach manages to achieve good results over a wide range of datasets and tasks, the improvements don’t seem significant enough. For the base-to-base/base-to-new results in Table 1, the gain in performances is compared to CoOp but not the second-best method PromptSRC (which can be misleading). On average, OrthSR only outperforms PromptSRC by 0.05% in terms of harmonic mean. For other tasks, the improvement in performance seems even slighter, which heavily weakens the contribution of the paper.

3. Excessive Training Cost:

While the paper claims at first hand that part of the focus is on efficient fine-tuning, the training cost inevitably increases when both branches of the CLIP encoder are involved, as is also shown in Table 5. Even though the counterpoint is that OrthSR achieves the highest inference speed, the number of training parameters is more than 10 times over the next most costly method, and more than 1000 times over baseline CoOp.

4. Poor Writing:

At times, the paper contains over-convoluted sentences with grammatical and even spelling mistakes, which makes it difficult for readers to ignore.

1. The proposed method introduces a significantly larger number of trainable parameters (10x–1000x more) compared to other methods, yet achieves only limited performance improvements, raising questions about its overall cost-effectiveness. Additionally, could you report other efficiency metrics, such as training time and FLOPs, for the proposed method compared to other baselines?
2. The performance improvements are marginal. For example, in the base-to-new generalization evaluation on certain datasets, the proposed approach underperforms compared to PromptSRC. The performance gains on two other tasks are also limited.
3. The motivation for introducing the attentive CutOut is unclear, and its connection to other components of the proposed method is not well explained.
4. A more detailed ablation study would be helpful. For example, how does the performance change when applying only the attentive CutOut?
5. Given the complexity of the proposed method, its reproducibility is limited. It is recommended that the authors provide the code implementation for this work.

Minor issues:

1. Line 31, classficiation -> classification.
2. Lines 303-306, kd -> kl?
3. Some of the notations in the methods section could be clarified. For instance, what is $f_t(:w)$ in Equation (1)? Additionally, could you explain what $k$ represents in $c_k$?
4. Line 279, the authors mention that “We empirically observe that this approximation results in instability of the fine-tuning”. Are there some related experiments? 5.Could you please clarify what the last row in Table 1 represents? Which baseline are you using for the performance gain comparison?