Progressive Parameter Efficient Transfer Learning for Semantic Segmentation

[R4-W5] Evaluation with More Backbones and Heads

[R4-A5]: As suggested, we conducted additional comparisons in the DINOv2 + Mask2Former setting with the counterparts on VOC2012 and CityScapes. The results are presented below. From the results, we observe that ProPETL continues to outperform the counterparts in this setting. We have included results from additional settings in the revised version to more effectively demonstrate the superiority of ProPETL.

[R4-W6]: Comparison to Rein

[R4-A6]: We compared ProPETL with Rein under the following semantic segmentation settings:

1. Cross-domain evaluation: GTAV (Train) → CityScapes (Test), as utilized in the Rein literature.
2. In-domain evaluation: CityScapes (Train) → CityScapes (Test), as presented in our paper.

In the cross-domain setting, ProPETL outperforms Rein (67.89 vs. 66.4, as noted in [R4-A1]). For the in-domain setting, since Rein did not report results, we conducted experiments using a supervised pre-trained ViT-Large backbone with a SETR [b] head. As shown in the table below, ProPETL achieves a 1.23% improvement in mIoU over Rein on CityScapes. These results demonstrate that ProPETL consistently outperforms Rein across both settings, underscoring its effectiveness.

[b] Zheng et al. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In CVPR 2021.

[R4-W7]: Failed examples, limitations and future work

[R4-A7]:

• Failure Cases: We have revised the manuscript to include an analysis of failure cases. As illustrated in Figure 9 in the appendix, ProPETL encounters challenges with segmentation redundancy in scenarios involving object occlusion (fifth row) and high foreground-background similarity (last row). Addressing these issues may involve optimizing intermediate tasks based on the data distribution characteristics.

• Limitations and Future Work: Looking ahead, several avenues merit further investigation. On one hand, while the midstream adaptation phase enhances fine-tuning performance, it does increase the overall computational cost of training. Exploring methods to shorten the global fine-tuning process and further improve fine-tuning efficiency remains an open challenge. On the other hand, the current design of intermediate tasks primarily relies on empirical studies. Developing dynamic approaches for designing intermediate tasks and generalizing ProPETL to accommodate a broader range of pretrained models and downstream scenarios are promising directions for future research.

[R4-W8]: Figure 1.

[R4-A8]: We have refined Figures 1(a) and 1(b) in the revised manuscript for improved clarity. And we will try our best to proofread the manuscript in the final version.

[R4-W9]: Code Release

[R4-A9]: We will release the code and checkpoints to ensure reproducibility and will make every effort to assist the community in reproducing the results upon acceptance.

[R4-W1]: Cross Scenerio Tasks

[R4-A1]: As suggested, we conducted a comparison with Full Fine-tuning, AdaptFormer, and Rein under domain generalization settings. DINOv2 and Mask2Former were utilized as backbones, following the official implementations of Rein[a]. The comparison was performed under GTAV → Cityscapes setup. The results demonstrate that ProPETL outperforms all counterparts with comparable parameters, underscoring its strong generalization capability in cross-scenario tasks.

We have included this analysis in the revised version to provide a more comprehensive evaluation of ProPETL's generalization ability.

[a] Wei et al. Stronger, Fewer, & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentation. In CVPR 2024.

[R4-W2]: Descriptions

[R4-A2]:

• Overall Training and Inference Process: The ProPETL training pipeline consists of two phases: midstream adaptation and downstream fine-tuning. In the midstream adaptation phase, we leverage image-level, multi-label intermediate tasks to train the adaptation module. During downstream fine-tuning, we adjust the adaptation module using the DSA strategy, followed by joint training with the segmentation head on the target segmentation task. For inference, the trained adaptation module, combined with the pre-trained backbone, extracts feature maps from input images, which are then fed to the segmentation head to generate the final segmentation results.

• Design of GPA's Architecture: The adaptation module in GPA follows a bottleneck architecture consisting of a down-projection layer, a ReLU activation, and an up-projection layer. Individual adaptation modules are inserted into each transformer layer. The intuition behind the design is to allow GPA to pre-initialize adaptation module parameters through optimization on intermediate tasks before downstream fine-tuning.

We will include more detailed descriptions in the final version and carefully proofread the manuscript to ensure clarity and precision.

[R4-W3]: Corresponding of Supervision Diversity and Perception Granularity

[R4-A3]: The parameter $\alpha$ (which controls supervision diversity) and the pooling window size (which controls perceptual granularity) are independently adjustable. There is no direct correspondence between them, as they govern different aspects of the intermediate task. We have revised the description in the experimental section to prevent any potential misunderstanding.

[R4-W4] Computation Cost

[R4-A4]: Please refer to [R2-A2] for a detailed comparison and analysis. We have included the computation cost comparison in the revised version.

We sincerely appreciate Reviewer 3 (fzgQ) for the positive and insightful feedback on our work. Below, we provide detailed responses to address your concerns.

[R3-W1]: Fixed Reduction Ratio

[R3-A1]: The primary goal of our work is to bridge the gap between pre-training and downstream tasks through a progressive adaptation process while maintaining a low parameter overhead. Thus, the role of $r$ is more to regulate the learnable parameters, mitigating the trivial effects of parameter fluctuations while achieving better trade-off between performance and parameter overhead. Regarding your concern that a fixed $r$ might result in the loss of critical feature information, we evaluate the performance under different values of $r$, and the results are presented below.

From the results, we observe that decreasing $r$ does not yield significant improvements in performance but does incur additional parameter overhead. This suggests that directly decreasing $r$ (add more parameters) may not address the bottleneck in the fine-tuning process for segmentation scenarios. Instead, the progressive paradigm appears to be more critical in this context. Therefore, we retain the original settings for their higher efficiency.

[R3-W2]: Theoretical Support of Intermediate Task Designation.

[R3-A2]: We appreciate the reviewer's insightful observation. Here, we would like to share some of our theoretical insights into the rationale behind our intermediate task design. Considering the standard formulation of the pixel-wise cross-entropy loss, $L_{\rm piexl-wise \ CE}$ for downstream segmentation tasks:

$$L_{\rm piexl-wise \ CE}=-\frac{1}{HW}\sum_{i=1}^{HW}\sum_{c=1}^{C}y_{i,c}\log(\hat{p}_{i,c}),$$

where $H$ and $W$ represent the height and width of the input, $C$ is the total number of classes, and $y_i$ is the one-hot encoded label vector for pixel $i$, with $y_{i,c} = 1$ if pixel $i$ belongs to class $c$. The term $\hat{p}_{i,c}$ denotes the predicted probability (post-softmax) for class $c$.

The perceptual granularity in intermediate task design adjusts the perception window sizes, thereby altering the outer loop, which governs the global supervision intensity. On the other hand, changes in supervision diversity impact the inner loop, determining the number of classes to classify. By combining these two dimensions, we can effectively interpolate between image-level classification and pixel-level segmentation tasks. This creates a spectrum of intermediate tasks that bridge the gap between pretraining and downstream fine-tuning. We evaluated typical combinations of these factors in the main paper to assess the effectiveness of midstream tasks.

[R3-Q1] Decoupled Structured Adaption

[R3-A3]: As suggested, we conducted an additional quantitative comparison between simultaneous tuning and progressive tuning. The results are presented below. While simultaneous tuning offers larger optimization spaces, it is more challenging to achieve superior results due to the issue of knowledge forgetting, as illustrated in Figure 7 of the main paper.

Please feel free to let us know if you have any further questions.

[R2-Q1]: Performance with Smaller/Less Powerful Pretrained Models

[R2-A3]: As suggested, we validated ProPETL using backbones with varying sizes and pretrained strategy. The results are presented below.

For the evaluation of model size, we compared Swin-Transformers of various scales, ranging from small to large, on the Pascal VOC 2012 dataset. The results indicate that ProPETL consistently outperforms the baseline across all settings. Furthermore, as the model size increases, the benefits of the midstream adaptation process become more pronounced, resulting in greater performance improvements.

For the evaluation of model capacity, we have assessed the generalization of ProPETL on models pretrained using MAE, as shown in Table 4 of the main paper. To further evaluate generalization, we employed the ViT-Base pretrained with MoCo-v3 as the backbone and conducted comparisons with full fine-tuning and state-of-the-art AdaptFormer. The results demonstrate that ProPETL significantly outperforms the baseline and linear probing by a large margin. Combining these findings with the evaluations in Tables 1 and 4 of the main paper, we conclude that ProPETL consistently delivers improvements across pretrained backbones of varying scales and capacities.

We have included these results in the revised version to provide a more comprehensive evaluation of ProPETL.

[R2-Q2]: Computation Cost and Training Time

[R2-A4]: Please kindly refer to [R2-A2] for a detailed comparsion and analysis.

Please feel free to let us know if you have any further questions or require additional clarifications.

We sincerely appreciate Reviewer 2 (GSCi) for the positive and insightful feedback on our work. Below, we provide detailed responses to address your concerns.

[R2-W1] Complexity and Generalization.

[R2-A1]: The proposed ProPETL introduces an additional phase into the downstream fine-tuning process without affecting inference efficiency, which aligns with the more common application scenario. A detailed comparison is provided in [R2-A2]. Additionally, the global computational cost is comparable to state-of-the-art methods, while achieving significantly superior performance on downstream applications.

In this paper, we aim to thoroughly investigate the degradation of fine-tuning performance in segmentation tasks. Our key insight is that semantic perceptual granularity shifts necessitate larger adjustments. It is worth noting that our primary objective is not to determine the optimal combination of perceptual granularity and supervision diversity, but rather to highlight that introducing midstream adaptation effectively bridges this gap and improves downstream performance. Consequently, we evaluate typical configurations of these two factors. In our experiments, we observed several trends in intermediate task design. Greater supervision diversity ($\alpha=1$) consistently leads to improved performance across all settings, suggesting that diverse supervision in the midstream adaptation phase better equips the model for downstream fine-grained dense predictions, as required in segmentation scenarios. Regarding perceptual granularity, performance varies across datasets, potentially influenced by the inductive biases of the downstream datasets. However, larger perceptual granularity generally delivers better results compared to patch-level settings ($s=32$).

[R2-W2]: Computation cost.

[R2-A2]: As suggested, we compare the proposed ProPETL with its counterparts in terms of computational cost. The results are presented below.

From the results, we observe that the inference time, workload, and peak GPU memory usage of ProPETL are comparable to state-of-the-art methods. Its only drawback is a slightly higher training time due to the additional phases. However, this training process is performed only once per fine-tuning session and delivers significantly superior performance in downstream applications. To further enhance the practicality of ProPETL, we introduce a lightweight version that reduces the duration of the second phase (i.e., downstream fine-tuning) by half. This adjustment results in a lower overall training time compared to all counterparts while still achieving significantly better performance in downstream tasks, underscoring the superiority of our approach.

We have included this comparison in the revised version to provide a more comprehensive evaluation of the prodposed ProPETL.

We sincerely appreciate Reviewer 1 (4Nue) for the positive and insightful feedback on our work. Below, we provide detailed responses to address your concerns.

[R1-W1]: Observations for SAM.

[R1-A1]: As suggested by R1, we selected two representative downstream tasks—classification (CIFAR-100) and segmentation (Cityscapes)—to examine the fine-tuning process on the segmentation-based pretrained model. Specifically, we calculated the Euclidean distance for SAM before and after fine-tuning. The results are presented below.

From the results, we observe that the effect of the granularity gap on parameter adjustment remains valid in this scenario. When fine-tuning the segmentation-pretrained SAM for downstream segmentation tasks (CityScapes), the magnitude of parameter adjustment is significantly smaller compared to fine-tuning for the classification task. This indicates that the perceptual granularity gap necessitates larger parameter adjustments. These observations further support our insights and complement the findings of the previous experiments.

To furtherly evaluate the proposed ProPETL in this scenerio, the results are illustrated as bellow.

From this comparison, we observe that the progressive adaptation process also enhances the fine-tuning performance for segmentation tasks, regardless of whether the backbone is pretrained for classification or segmentation. This further demonstrates the effectiveness of the proposed approach. We will include these additional comparisons in the final version to provide a more comprehensive evaluation of the proposed methods.

[R1-W2-Q1]: The Necessity of Midstream Adaptation, Its Relationship with Data Augmentation, and Its Integration with Fine-Tuning

[R1-A2]: Building on the observation that a larger perceptual granularity gap between pretrained models and downstream tasks necessitates more substantial adjustments, introducing a midstream step to bridge this gap during the fine-tuning process emerges as a natural solution. This paradigm has proven effective in applications such as [a] and [b]. However, this progressive approach remains largely unexplored in the field of PETL. To address this limitation, we conducted a detailed analysis of midstream task designations and progressive strategies in this work. The experiments presented in Tables 2 and 3, as well as Figure 6 in the main paper, validate the effectiveness of midstream adaptation.

We respectfully differentiate the proposed midstream adaptation from classical data augmentation, as they address different aspects of the training process. Data augmentation perturbs the input data, encouraging the model to produce consistent predictions. In contrast, midstream adaptation retains the same inputs but modifies the annotations by incorporating different granularities and diverse forms of supervision. To further compare the proposed midstream adaptation with data augmentations and evaluate its integration with fine-tuning, we employ AdaptFormer as the baseline and conduct segmentation fine-tuning on the VOC2012 dataset. The results are presented below.

The results indicate that while data augmentations slightly improve mAcc, they lead to a drop in mIoU, as they fail to address the granularity gap due to their sole focus on modifying the input images. In contrast, when ProPETL is employed, the performance significantly improves to 86.11 mIoU and 92.08 mAcc, highlighting the differences between midstream adaptation and data augmentation. Furthermore, integrating midstream adaptation with the fine-tuning process yields better performance than the baseline but still falls short of the default ProPETL. This discrepancy arises because midstream and downstream tasks emphasize different perceptual granularities, potentially causing conflicts that hinder model convergence and result in performance degradation.

[a] Weinshall et al. Curriculum learning by transfer learning: Theory and experiments with deep networks. In ICML 2018.

[b] Hsu et al. Progressive domain adaptation for object detection. In WACV 2020.

Please feel free to let us know if you have any further questions or require additional clarifications.