Double-Filter: Efficient Fine-tuning of Pre-trained Vision-Language Models via Patch&Layer Filtering

We thank the reviewer for their very detailed and instructive feedback. We are very happy that you recognize our motivation and work and give positive support. Regarding your suggestions and concerns, we will respond one by one below, hoping to answer your questions.

Q1. “YOLO's object detection may be inaccurate, leading to incorrect foreground-background segmentation and potentially affecting the patch selection process.”

Thanks for your comment. Indeed, the situation you mentioned is possible. But this is one of the motivations behind our decision to remove redundancy in the foreground and background separately and retain a certain proportion of patches respectively. On the one hand, our attention mechanism is based on the global attention distribution, so even if there are missed or wrong detections in the foreground, our IPF method can retain these important entity areas in the background (we assume they are important). Therefore, in this case, it can maintain the same global salient object attention ability as EViT(Liangetal.,2022). On the other hand, since we distinguish the foreground and background, we can pay more attention to the background than EViT, which only focuses on the salient area, even if there is no clear semantics, it is very important for the logical integrity of the image context expression.

Q2. “The paper lacks a detailed analysis of YOLO's parameter count and FLOPs, which is important for assessing its computational cost in the proposed method.”

Thank you for the helpful suggestion. Specifically, YOLOv8 used in our IPF module requires approximately 4.5G FLOPs per forward pass on one RTX 3090Ti. Taking the METER model as an example, it requires a total of about 100G FLOPs cost in per forward pass, of which each transformer Block in the interaction network requires approximately 5G FLOPs. Therefore, YOLOs accounts for less than 5% of the total, which is approximately equal to the cost of one Block. We will revise Section 3.3 and Table 1 in the new paper to explicitly include this small cost to provide a complete picture of all forward computations.

Q3. “Why does the paper not consider addressing redundancy in the text modality?”

Thank you for your insightful question and we appreciate the opportunity to clarify this aspect of our work. Our study focuses primarily on image redundancy because images account for the majority of computational overhead in Vision-Language Pretraining (VLP) models. As discussed in Section 3.3 (Lines 259–264) of our paper, textual redundancy is considerably lower than that of image patches, making image redundancy the more critical bottleneck to address. Tasks commonly used in VLP, such as image captioning, VQA, and cross-modal retrieval,etc., typically involve short text sequences. For example, in the METER and ViLT models we adopt, the maximum text length is only 40–50 tokens, whereas ViLT-B/32 processes over 240 image patches—meaning text accounts for less than 20% of the total input length. Similar observations are also reported in [1, 2]. Moreover, since the computational cost (FLOPs) of transformer-based models scales linearly with sequence length, the relatively short length of text implies that the FLOPs consumed by the text modality are inherently much lower. Therefore, we concentrate on optimizing the visual modality, where redundancy is more significant and computational savings are more impactful.

References:

[1] Yang, Senqiao, et al. "Visionzip: Longer is better but not necessary in vision language models." arXiv preprint arXiv:2412.04467 (2024).

[2] Kim, Wonjae, Bokyung Son, and Ildoo Kim. "ViLT: Vision-and-language transformer without convolution or region supervision." International Conference on Machine Learning, PMLR, 2021.

Q4．A minor but important point: I noticed that the paper inconsistently uses "FLOPS" and "FLOPs" to refer to floating-point operations.

Thank you for pointing this out. We will standardize the terminology throughout the paper, consistently using "FLOPs" to ensure clarity and accuracy. And we will carefully proofread the full text to ensure there are no typos.

We hope that the above responses could address the reviewers' concerns and questions.

We thank the Reviewer very much for the kind words, for the interest in our research activities, and for the very insightful comments.

W1: “Potential weaknesses include an increased system complexity due to the integration of a YOLO detector and genetic algorithm, and limited exploration of the method’s generalizability beyond VQA and NLVR.”

Thanks for the comment. For the system complexity of the integration of a YOLO detector and genetic algorithm, as shown in Table 1 and Table 2, after introducing YOLO-based IPF and AGA-based ALF, our Double-Filter has significantly reduced the calculation of FLPOs, and the inference time has been greatly improved. Although the model introduces more algorithms, the inference efficiency of the VLP model has been significantly improved.

For the exploration of the method’s generalizability to more tasks, we further experimented on more complex image and text retrieval tasks by renting a larger GPU. Due to time limits, the response Table reports our Double-Filter with same setting as Table 1 in the paper compared to the mainstream methods.

On the cross-modal retrieval task, our proposed Double-Filter also shows the lowest computational complexity among existing PEFT methods, and maintains competitive performance. This further demonstrates the task generalization ability of our proposed Double-Filter method.

Q1: “Could you elaborate on your choice of a genetic algorithm for the fine-grained Architecture Layer Filter?”

Thanks. As description in Section 3.2, we propose an Adjustable Genetic Algorithm (AGA) for fine-grained architecture filtering, approximating an optimal configuration. Based on the traditional genetic algorithm, it introduced adaptive adjustment for custom number of filtering layers, so as to better perform adaptive training and deployment capabilities on different devices. Each model configuration is encoded as a chromosome, with genetic operations (fitness evaluation, crossover, and mutation) guiding optimization across generations. The fitness function balances loss on sampled datasets, ensuring optimal layer reduction. Crossover swaps genes between parents, while mutation adjusts gene values to maintain a fixed number of replaced layers.

Additionally, as mentioned in Appendix A.1.2 (Lines 551-557), our loss calculations in the fitness function were based on 100 batches of training data (equivalent to 2 epochs), and losses were recorded over 10 validation batches. The search cost of GA is equivalent to adding 2 epochs to the training phase, but DAS requires 3 epochs and requires full validation set validation. So the GA cost is still more efficient than the comparison method.

Moreover, please refer to our response to Q2, which further discusses optimization quality and stability.

Q2: “Given that GAs are heuristic and do not always guarantee an optimal result and requires hyperparameter tuning, did you consider alternative optimization methods that might offer stronger guarantees on the quality or stability of the optimization outcome?”

Thanks for the insightful comment. As we said in Footnote2 on Page 5, identifying an absolute optimal structure is NP-hard, rendering exhaustive search impractical. So as mentioned in Section 3.2 (line 252), "The AGA aims to identify the approximate optimal reduction with L filtered layers."

Comparing with other searching algorithms, genetic algorithm uses population search to avoid falling into the local optimal solution, and has a greater probability to find the global optimal solution. And another reason we choose GA-based model is that genetic algorithms can be well parallelized and improve computational efficiency, because of the independent evaluation among individuals.

To offer stronger guarantees on the quality or stability of the optimization outcome, the "Elite Strategy" is adopted in step 11 of Algorithm 2 to ensure that the optimal solution of the current generation is not lost during each generation update, thereby ensuring the quality of the search. We design the mutation operation to satisfy the constraint of the genetic algorithm while diversifying each generation effectively.

We hope that the above responses could address the reviewers' concerns and questions.

Thanks for your valuable and constructive comments. We appreciate your recognition of our work's innovative designs, evaluation criteria, and insightful analysis. Below, we address your concerns and questions individually:

Q1: “In IPF section, the paper mentions sparsity ρ ∈ (0, 1). Is this sparsity a hyperparameter, and if so, how is it set or optimized during the training process? Could the authors provide more details on how this parameter is determined?”

Thanks for your detailed suggestion. The sparsity parameter (ρ) in the image patch filter is actually a hyperparameter that controls the filtering ratio of the image patches. The larger the sparsity parameter, the greater the filtering degree. In fact, in Table 3 of the paper, we have conducted ablation experiments by setting the sparsity parameter (ρ) differently to achieve a balance between efficiency and performance. We will further clarify this in the final version to avoid confusion.

Q2: “The authors claim that YOLOv8’s cost is much lower than other modules, but no clear evidence is provided. Please estimate the forward pass cost of this module,...”

Thank you for the helpful suggestion. In our original analysis, we omitted the YOLOv8 cost because it is negligible compared to the overall model computation. However, for completeness, we now explicitly to calculate it. Specifically, YOLOv8 used in our IPF module requires approximately 4.5 G FLOPs per forward pass on one RTX 3090Ti. For the ViT-based encoder, taking the METER model as an example, it requires a total of about 100G FLOPs cost in per forward pass, of which each transformer Block in the interaction network requires approximately 5G FLOPs. Therefore, YOLOs accounts for less than 5% of the total, which is approximately equal to the cost of one Block.

We will revise Section 3.3 and Table 1 to explicitly include this small cost to provide a complete picture of all forward computations.

In addition, as shown in Table 1 and Table 3, even considering the cost of YOLO, our Double-Filter methods significantly reduces the computational cost when fine-tuning the VLP model, especially for models with complex interactions such as the METER model.

Q3. “Why choose METER and ViLT instead of other VLP models as the base models for experiments? The author needs to give a clearer explanation and reason.”

Thanks for the insightful suggestion. Our choice of METER and ViLT as baselines was based on the following considerations:

(1) METER and ViLT are the primary baselines for our comparison methods. To ensure a fair and meaningful evaluation, we followed the standard practice of using the same VLP models as our comparison baselines.

(2) METER and ViLT represent two distinct VLP architectures. METER employs complex cross-modal fusion mechanisms, while ViLT maintains independent unimodal encoding. Our experimental results demonstrate that our Double-Filter achieves more significant efficiency improvements under the complex METER framework. This suggests that Double-Filter has the potential to be even more effective when applied to more complex VLP models.

Given these considerations, we choose METER and ViLT. Nonetheless, we acknowledge the importance of further validation on additional VLP models and will explore this in future work.

Q4. “In the Fitness Function, why must ( \beta ) be greater than half of the maximum summed losses? How does this choice impact the optimization process?”

Sorry for the confusion. In this paper, the role of $\beta$ is precisely to transform fitness values into the positive domain. In genetic algorithms, roulette wheel selection necessitates that the fitness scores generated by the fitness function be positive, as its core mechanism depends on transforming fitness values into non-negative and normalized probability distributions. If fitness values are negative, it not only leads to calculated selection probabilities being negative, which contradicts the fundamental principles of probability, but may also trigger division-by-zero errors or render probability calculations invalid when the sum of fitness values is zero or negative. Consequently, by employing a translation method to shift all fitness values into the positive domain, the relative ranking of individuals is maintained, thereby ensuring the algorithm's effective and reliable operation. We will clearly these discussion in the new revision.

Q5. “On page 6, in the left column, there are two formulas that appear to be excessively long.”

Thank you for the detailed suggestion! We'll simplify or break these formulas down into clearer segments to enhance readability.

We thank the reviewer for the thorough review, acknowledging our contributions, and providing instructive suggestions. We respond to each weakness (W*) or question (Q*) below.

W1: “... explore the impact on more complex multimodal tasks beyond VQA and NLVR, ...”

Thanks for the instructive suggestion. We conducted VQA and NLVR experiments on one NVIDIA RTX 3090Ti, but the retrieval task requires many negative samples, making it difficult to maintain the same batch size compared to existing methods.

To address the concerns, we rented a larger GPU and conducted generalization experiments on Flickr30K for image-text retrieval. The response Table reports our Double-Filter with same setting as Table 1 in the paper compared to the mainstream methods.

Our Double-Filter achieves the lowest computational complexity among PEFT methods and maintains competitive performance in image-text retrieval.

W2: “The authors do not discuss the computational cost of running the GA, which ...”

Thanks for the detailed suggestion. As mentioned in Appendix A.1.2 (Lines 551-557), our loss calculations in the fitness function were based on 100 batches of training data (equivalent to 2 epochs), and losses were recorded over 10 validation batches. The search cost of GA is equivalent to adding 2 epochs to the training phase, but DAS takes 3 epochs and requires full validation set validation. So the GA cost is still lower than the comparison method.

W3: “A more detailed discussion of the limitations of the proposed method.”

Thanks for the detailed suggestion. We will further discuss detailed limitations. On the one hand, our method relies on pre-trained target detectors, but it still adds additional inference time; on the other hand, we use genetic algorithms to search for the best subnet, which has a long search time in the training stage and is prone to fall into the limitation of local optima. We will add the discussion in the final version.

Q1: “Have the authors tested Double-Filter on other VLP models.”

Thanks for the instructive comment. We choose METER and ViLT as baselines because:

(1) METER and ViLT are the primary baselines for our comparison methods. To ensure a fair evaluation, we followed the standard practice of using the same VLP models as our baselines.

(2) METER and ViLT represent two mainstream VLP architectures, and others generally follow these architectures. METER employs complex cross-modal fusion, while ViLT maintains independent unimodal encoding. Our results show that Double-Filter achieves notable efficiency improvements, especially under METER, suggesting its potential for more complex VLP models.

Q2: “Have the authors considered the impact of Double-Filter on inference time and memory usage.”

Thanks. In fact, as shown in Table 2 (Line 332), we have compared the inference speed of Double-Filter with other methods. Our results demonstrate that Double-Filter enables processing more samples per unit time, indicating a clear efficiency improvement. The following table provides additional memory consumption (inferencing with batch size of 1) to further illustrate the lower memory of Double-Filter.

Q3: “Have the authors explored other object detection methods.”

Thank you for highlighting this vital direction. We employed YOLOv8 because the YOLOs are the most mainstream object detectors, which can better balance effectiveness and efficiency of detection. We further test EfficientDet[1], which need more computing resources. Due to the limited rebuttal time, we only compared the IPF filtering overlap among detectors, and found that over 96% of filtered patches remained consistent, so we believe that this has little impact on the final patch retention. Future work will explore alternative methods, including lightweight offline pre-training.

[1] Tan, Mingxing, Ruoming Pang, and Quoc V. Le. "EfficientDet:Scalable and efficient object detection." CVPR. 2020.

Thanks for the encouraging feedback and suggestions. However, we argue that some concerns were addressed in our paper, so we clarify them again here. We address the main weaknesses(W*) and questions(Q*) below:

W1: “YOLO's computational cost is not accounted for in efficiency metrics ,....”

Clarification: We would like to clarify that our paper explicitly states that our experiments take YOLO’s FLOPs into account when calculating the model FLOPs, in the corresponding captions of Table 1 (Line 278) and Table 3(Line 332), e.g. “The FLOPs of Double-Filter contain the YOLO detector.”. We compare complete Double-Filter with others in Table 1 and show the FLOPs reduction statistics via the diversity IPF in Table 3. We hope this clarification resolves the reviewer’s concern.

W2: “The experimental design lacks diversity, focusing solely with fixed sparsity ratios (e.g., 60% patch filtering).”

First, we clarify that Table 3 and Figure 3 present ablation studies on various sparsity ratios, which is inconsistent with the reviewer's claim of a fixed ratio.

Second, exhaustively testing all ratio combinations would demand substantial computational resources. To balance performance and efficiency, we evaluate four representative ratios—40%, 50%, 60%, and 70%. This approach ensures a comprehensive yet feasible analysis and follows a widely adopted experimental methodology.

W3: “YOLO’s inference time likely outweighs the savings from patch pruning.”

As shown in Table 2, YOLO’s inference time is included in Double-Filter. Despite this, it still improves overall inference speed. We will clarify this in the final version.

W4: “The Appendix provides incomplete details...”

The main parameters of the genetic algorithm were given. We guess the concerns lie in YOLO. We clarify that our pipeline does not rely on additional settings of YOLO, and directly uses the pre-trained YOLOv8 (Reisetal.,2023). Therefore, it does not affect our reproducibility. We will add more details in the final version.

W5: “The lack of benchmarking against LayerDrop-style dynamic inference ...”

LayerDrop-style methods typically required modifications during the pre-training stage to enable adaptive inference, whereas our focus lies in fine-tuning pre-trained VLP models without altering the upstream pre-training process. This makes the comparison less aligned in terms of scope—note that DAS (Wu et al., 2023) also does not compare with such methods. Moreover, unlike dynamic methods that often require loading the entire model and selectively dropping layers, our method only loads a searched subnetwork, reducing memory and computational overhead.

Q1: “How does YOLO’s computational cost compare to the FLOP savings from patch/layer pruning?”

Taking METER as an example, as shown in Table 1, our Double-Filter can reduce computational costs by 21.18G FLOPs. The computational costs consider the YOLO’s FLOPs (about 4.5G FLOPs), so there are about 26G FLOPs savings from IPF and ALF. Additionally, Table 3 also shows the detailed computational costs including YOLO’s FLOPs when applying different filter ratios for IPF. The FLOPs saved by image patch filtering far exceeds the introduction of YOLO.

Q2: “Can the pipeline function without YOLO for fair comparison to prior work?”

YOLO effectively distinguishes foreground and background without complex design, enhancing versatility. In future work, we will further explore how to replace YOLO, but this will undoubtedly require a tailored design, and the versatility may be reduced. Additionally, as we clarified in W1, our DF included YOLO's cost when comparing with others, so they are fair comparisons.

Q3: “Why were retrieval tasks excluded, given their importance for VLP benchmarking?”

Sorry for the confusion. Due to resource limitations, we conducted VQA and NLVR experiments on an NVIDIA RTX 3090Ti. However, retrieval require numerous negative samples, making it difficult to maintain the same batch size as in prior works (e.g., DAS used NVIDIA Tesla A100 GPU).

To address the concerns, we rented a larger GPU for image-text retrieval task on Flickr30K.

Our model for retrieval task still maintains a high efficiency while ensuring the performance with the same setting as Table 1.