Replacement Learning: Training Vision Tasks with Fewer Learnable Parameters

1. The primary concern with the paper is that the experiments are conducted by training networks from scratch. Transfer learning boosts the performance of classification tasks, for instance in the official vit paper[1], VIT B/16 upon transfer learning on CIFAR10 reached an accuracy of 98.13% while the paper achieves only 72.86%, therefore it is important to perform an experiment for examining the effect of replacement learning when transfer learnt. One possible reason why the paper omits trying transfer learning can be initialization of replaced parameters. Parameter Integration $a * \theta_{i-1}+ b* \theta_{i+1}$ is a linear model, this allows approximating a,b values for pre-trained layers in few steps.

2. The paper only shows results on classification, effect of replacement learning on downstream tasks such as object detection and segmentation would strengthen the approach.

3. The flow of gradients is not in a singular direction during replacment learning as suggested by Eq. 15 where $\delta_{i}$ is used for gradient computation of $\delta_{i+1}$. This could cause issues such as vanishing or exploding gradients and must be given a look at.

4. While the experiments show better time per epoch, the effect of replacement learning on convergence must be studied. This is important as time for 1 epoch can be misleading when the number of epochs to converge is significantly higher than backpropagation.

5. While the paper addresses that only MSA layers were chosen for replacement, where $\theta_{i-1}$ and $\theta_{i+1}$ also MSA layers? This is important as [2] does not guarantee correlation between MSA and MLP layers.

Nitpick: some figure references are wrong, Figures in the ablation display figure numbers 4.3.1, 4.3.2 and 4.3.3 which need to be corrected to figure 3,4,5. Usually it is the placement of caption and label in \begin{figure} that causes this issue.

References: [1] Dosovitskiy, Alexey. "An image is worth 16x16 words: Transformers for image recognition at scale." arXiv preprint arXiv:2010.11929 (2020). [2] Venkataramanan, Shashanka, et al. "Skip-attention: Improving vision transformers by paying less attention." arXiv preprint arXiv:2301.02240 (2023).

1. Title: The authors use "Visual Tasks" in the title. "Visual Tasks" include multiple tasks such as classification, detection, segmentation, etc., but it seems that the paper is only validated on the classification task. I suggest adding other tasks to the paper, as has been done in several recent PEFT works [1-3]. [1] 1% vs 100%: Parameter-efficient low rank adapter for dense predictions. [2] Pro-tuning: Unified prompt tuning for vision tasks. [3] Adapter is all you need for tuning visual tasks. [4] Parameter-efficient is not sufficient: Exploring parameter, memory, and time efficient adapter tuning for dense predictions.

2. Related work: the authors perhaps left out some of the most recent work of parameter-efficient fine tuning (PEFT).

3. Experiments: (1) Experiments are performed only on classification tasks; (2) More parameter-efficient fine-tuning methods are available for comparison.

I would consider increasing the score if the authors could provide more convincing comparative experiments.

1. Figure 1 doesn’t make any sense—using very large models on very small datasets. It would be more meaningful to test on a larger dataset like ImageNet, with different models of different scales for comparison.

2. what is the motivation of such a method?

3. There are several unsupported and arbitrary statements in the paper. For example, in Line 83, "Considering that parameters from adjacent layers, if solely derived from either shallow or deep layers, often fail to simultaneously enable frozen layers to excel in learning both local and global features" lacks justification. The relationship to local or global features is unclear here, as adjacent layers don’t inherently correspond to shallow or deep feature characteristics.

4. In Eq. 9, the parameters of the i-th layer are a linear combination of those from the previous and next layers. What’s the motivation behind? It’s also unclear why this setup would yield better performance, as shown in Table 1 and Table 2.

5. What is the motivation for providing detailed information about backpropagation? I don’t see any differences or novel contributions in this section.

6. In the experiments, I don’t understand how reducing the number of parameters in a model can consistently improve performance across different datasets and models. This seems completely unreasonable and doesn’t make sense at all, especially I didn't see anything can help with this.

7. BTW, the results on CIFAR-10, SVHN, and STL-10 aren’t reliable, as these datasets are too small to provide meaningful insights.

8. Why is k=4 chosen? There should be an ablation study to justify this choice. Additionally, why is there a frozen layer every k layers throughout the networks? I assume we can select layers.

9. What happens if the frozen layer is the last layer in a ResNet stage? How would parameter interpolation be handled in this case?

10. The experimental settings MUST have issues. All the results for the vanilla models are SIGNIFICANTLY lower than original papers. This calls into question the reliability of the results presented in the paper, potentially not only ImageNet.

11. Typo： feature maps can be found in Figure 4.3.1.

12. I cannot tell any essiential differences among the four images in Fig. 3.

1. The proposed method only marginally reduces the GPU memory consumption and training time compared to the baseline training method.

2. The paper did not compare the proposed method with any other parameter-efficient training methods, such as [3].

3. Although the paper discusses related works on alternative backpropagation methods and training utilizing surrounding layers, none of the related works are compared with the proposed methods in the experiments.

4. Parameter-efficient training methods [1, 2] are widely applied in fine-tuning pre-trained networks by selectively updating a small subset of model parameters, streamlining the adaptation process of pre-trained models, and facilitating rapid deployment across various domains. However, this paper only studies the setting for training-from-scratch.

[1] Zhang, Taolin, et al. "Parameter-efficient and memory-efficient tuning for vision transformer: a disentangled approach." ECCV 2024.

[2] He, Xuehai, et al. "Parameter-efficient model adaptation for vision transformers." AAAI 2023.

[3] Mostafa, Hesham, and Xin Wang. "Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization." ICML 2019.

I like the paper overall. However, would like to point some weaknesses which the authors have also mentioned in their limitations section:

1. While effective on image-based tasks, the approach has not yet been tested on other domains such as NLP or multimodal tasks, which limits its generalizability.
2. The paper could benefit from a more in-depth discussion of any limitations associated with freezing certain layers and its impact on long-term learning dependencies, especially in deeper networks.