Noise-Robustness Through Noise: A Framework combining Asymmetric LoRA with Poisoning MoE

Thank you very much for your recognition of the innovation and overall quality of our work. We are very impressed by your meticulous and insightful comments, through which we deeply sense your expectation for us to make this work better. The core of our work that you considered is absolutely correct, and almost all other reviewers agree with you. Below, we summarize and explain your concerns point by point:

1. Sharing and fine-tuning the A-matrix in both Stage I and Stage II raises the question of whether harmful information learned during the first stage (from noisy data) might be absorbed into the A-matrix, potentially leading to performance degradation.

Response: Thank you for your valuable comment! Here is our theoretical analysis regarding this question: Based on the further exploration of the knowledge of matrix A in the HydraLoRA paper, we believe that matrix A learns relatively language environment knowledge, which is very basic and universal, and very little affected by noise(a dataset with added noise can be analogized as a new domain dataset). In addition, we have conducted two experiments in the past few days to address your concerns. Firstly, we conducted a comprehensive visualization of the A and B parameters in LoPE to present the actual changes in each part of the process. Due to format limitations of the rebuttal, we will first provide the key visualization results, and we will include all the figures and a complete analysis in the camera-ready version. Our findings show that during the entire two-stage fine-tuning process, the position of matrix A in the visualization space changes very little, which is fully consistent with our previous conclusions in the manuscript, and also aligns with the HydraLoRA paper’s conclusion that matrix A represents general knowledge. And we further conducted experiments by freezing the matrix A obtained after the first stage. The table presents our new findings and reveals an interesting phenomenon: we observed a significant drop in accuracy. Through careful investigation, we identified the root cause and are happy to share and discuss it with you as follows:

Our analysis shows that freezing matrix A immediately after its initialization in the first stage will hinder convergence and reduce performance. This finding is consistent with observations from the ablation study in "LoRA: Low-Rank Adaptation of Large Language Models", which documented a similar effect. However, our freezing approach does not result in severe negative impacts, as we still have the second stage of training that unfreezes and normally trains matrix A, though freezing A in Stage I does significantly affect the speed of the first stage.

Based on these results, we draw two key conclusions:

First, as noted above, matrix A exhibits stability—its fundamental structure remains largely unchanged throughout training. Furthermore, as demonstrated in HydraLoRA, the general knowledge encoded in matrix A is resistant to noise-induced perturbations.

Second, and perhaps more importantly, our two-stage training paradigm actually enhances the robustness of matrix A. The dynamic learning process does more than just preserve its initial state; it reinforces its representation of noise while maintaining its core knowledge structure over both stages. This indicates that allowing matrix A to adapt during training, rather than freezing it prematurely, is crucial for achieving optimal noise robustness. 

2.HyNoIse is primarily a character-level noise and may not reflect real-world, high-complexity datasets. Is it possible for the authors to provide analysis of the performance in a real task?

Response: Thank you for your valuable comment! As described in Section 3.2.1, HyNoIse represents a comprehensive noise injection method that extends beyond character-level perturbations. It encompasses character-level, token-level, label-level, and structural-level typical noise patterns, which comprehensively cover the common anomalies we have surveyed in real-world scenarios, including continuous noise in the embedding level, designed to simulate diverse real-world noise conditions. I understand your concern about the performance of real-world tasks as we do. Therefore, our evaluation employs a carefully curated suite of benchmark datasets: MMLU, PIQA, SIQA, and GSM8K, collectively spanning mathematics, commonsense reasoning, chemistry, history, law, and numerous other domains. Here are a few examples to illustrate its properties: 

• MMLU (Massive Multitask Language Understanding) spans tasks across multiple domains, enabling comprehensive evaluation of models' language understanding capabilities across diverse knowledge areas.
• PIQA (Physical Interaction QA) is designed to evaluate models' ability to predict physical object interactions and commonsense physical understanding.
• SIQA (Social Interaction QA) serves as a benchmark for evaluating social commonsense reasoning and interpersonal intelligence.
• GSM8K provides grade school math problems requiring multi-step mathematical reasoning.

And we also added experiments on the tasks mentioned in the related work as follows:

The experimental results demonstrate that our method achieves superior performance across all evaluation metrics, indicating the generalizability of LoPE to real tasks.

We believe that these widely adopted benchmarks collectively provide a comprehensive assessment of our model's practical capabilities across different cognitive tasks and knowledge domains. In addition, we will add more performance and corresponding analysis of LoPE in specific downstream natural language processing tasks in the Camera-Ready version. Thanks again!

3.Missing hyperparameter analysis of β.

Response: Thank you for your valuable comment! We are deeply sorry that there is an ambiguous notation regarding β in Equation (3)... To clarify, β serves as a normalization factor rather than a hyperparameter. Since the sum of the subsequent coefficients exceeds 1, we introduce β to ensure that

thereby maintaining consistent scaling across all coefficients (the same principle applies to Equation (4)). This normalization ensures that all components adhere to the same scale. We will revise this notation and provide a clearer explanation in the revised manuscript.

4. The first paragraph of Section 1 lacks a relevant citation

Response: Thank you for your valuable comment! We have added the relevant citations to the introduction section as suggested. We greatly appreciate your guidance in this regard. We will carefully investigate and add other related work in the Camera-Ready version. If you believe there are other important works relevant to our research contributions that should be cited, we would be very grateful for your valuable suggestions. Thanks again!

5. For the without noise performance in Table 1, why does LoPE improve compared to HydraLoRA, if I understand correctly, LoPE lies mainly in the two stages of fine-tuning the noise data, but why would this improve the Orig dataset as well?

Response: Thank you for your valuable comment! Our primary motivation centers on handling data containing substantial noise. After completing the noisy data experiments, we recognized that even "original" datasets inherently contain noise, albeit potentially at lower levels. As documented in "Learning from noisy labels with deep neural networks: A survey" (cited in our manuscript), no real-world dataset achieves perfect cleanliness.

While our framework treats original data as "zero-noise" theoretically for comparison with our artificially noise-injected datasets, this represents a relative rather than an absolute characterization. The reality of large-scale datasets is that complete clean remains unattainable—an inherent limitation of current data collection and annotation practices. Consequently, LoPE demonstrates performance improvements even on unprocessed original datasets because it effectively handles the latent noise present in ostensibly "clean" data. This observation underscores the practical utility of our approach beyond artificial noise scenarios.

Thank you again for taking the time to review our response. We sincerely hope that our response can solve your concerns and better evaluate the comprehensive value of our work. Thanks again!

Thanks for your thorough response and the additional experiments. I've gone through your answers, the reviewer discussions, and the new results.

While your clarifications helped address some of my questions, I still have a few lingering concerns. The new findings about freezing matrix A after Stage 1 and the resulting performance drop are interesting and align with prior work, reinforcing that adaptation is key for convergence. That said, this doesn’t directly confirm whether harmful noise is being absorbed into matrix A. You mention that its "fundamental structure remains largely unchanged" and that it's "resistant to noise-induced perturbations," but these are more qualitative claims. The performance drop in the ablation study could stem from factors other than noise absorption, making it tricky to draw firm conclusions here. A more direct analysis or a stronger ablation study is welcomed.

As for the other points, your responses have cleared those up.

Thanks for your time and feedback, we deeply sense your expectation for us to make this work better. In this paper, we boldly explored the 'Noise Robustness Only Through Noise' approach and innovatively proposed LoPE. By combining the advantages of multi-stage differentiated training under the PEFT framework, we completely eliminate the requirement of data cleaning. Furthermore, the proposed LoPE can be widely applied to the parameter-efficient fine-tuning of large language models, enhancing its potential to improve model performance with limited computational resources.

Additionally, we consider that the proposed poisoning expert elimination strategy is not only applicable to noise-related tasks but also offers new insights for diverse scenario tasks that are challenging to model or solve directly.

In the main aspects, including motivation, paper quality, experimental results, and reproducibility, our work was fully approved by all other reviewers.

Through your comments, we noticed that there are some misunderstandings about our work. In the following, we will summarize and explain your concerns point by point in order to clarify them.

R1:

Thanks for your comment. In our experiments, Orig and Nois represent two independent experimental workflows, not stages within a single pipeline. Each follows the same full two-stage fine-tuning and evaluation process. Specifically, Orig is the original dataset (with inherent noise), while Nois augments it with 5% additional discrete noise to create a more challenging and comparable setting. The clean data (in stage 2) you mentioned actually corresponds to the datasets used in each setting. In contrast, the Stage 1 data generated via HyNoIse is additional noise specifically crafted to train the poisoning expert, which is distinct from the Nois setting created for a more challenging and comparable evaluation.

Whether fine-tuning on Orig or Nois, we follow the same mechanism:

Stage I: We use the fine-tuning dataset augmented via HyNoIse to exclusively train only the poisoning expert, while freezing other experts, ensuring that noise adaptation is only highly concentrated in it.

Stage II: We freeze the poisoning expert to preserve its noise adaptation and train the normal experts and router on the dataset without further HyNoIse (i.e., the Orig or Nois dataset itself). Consequently, the learned routing distribution increasingly assigns noisy samples to the poisoning expert and clean samples to others, achieving our intended noise isolation.

Crucially, we use the HyNoIse only in Stage I to train the poisoning expert; it's never used in Stage II. In the inference stage, we mask the poisoning expert, so the final output is generated only from the normal experts.

Regarding "Since the noisy data that synthetically generated from the clean data, and many studies on noise injection or data augmentation rely on clean data to synthesize noise, we consider that this does not diminish their effectiveness. “Data Augmentation Approaches in Natural Language Processing: A Survey” explicitly states that “adding noise to the original data while preserving label validity can improve model robustness.”

Thanks again for your comment! Please feel free to reach out for any further discussion.

R2:

Thanks for your comment. With PEFT paradigms becoming the mainstream framework for large language models, we recognize that HydraLoRA serves as the architectural foundation for many subsequent innovations, and we believe that building upon this architecture through novel designs should be encouraged, similar to how many works build upon the LoRA framework. Several of HydraLoRA-based works have been widely acknowledged by the community (e.g., TeamLoRA, ACL 2025). As mentioned below, our core contribution lies in proposing a novel, complementarity-inspired paradigm. We believe that it is not only applicable to noise-related tasks but also provides new insights for more scenarios that are difficult to model or solve directly.

R3:

Thanks for your comment. As demonstrated in the experiments of the HydraLoRA paper, higher ranks generally lead to better performance. However, increasing the rank also incurs higher parameter and computational costs, making it necessary to balance scalability and effectiveness. We believe that a similar trend holds for our LoPE framework.

We have added results for LoPE with ranks 8 and 16, and compared them with HydraLoRA at the same ranks. The observed performance trends remain consistent with our previous conclusions, and we also expect a similar trend to hold at higher ranks. † denotes methods fine-tuned on Nois.

Notably, HydraLoRA’s MMLU result (47.22%) was obtained using Databricks-Dolly-15k. However, we can't find a usable version in their repository. All of our experiments were fine-tuned on the Alpaca-52K dataset (see Section 4.1). We believe that differences in fine-tuning datasets naturally lead to variations in inference performance.

We will include both the results for ranks 4 and 8 as the main results in the camera-ready version, and provide a complete ablation study with corresponding analysis for different ranks in a later section. We thank you again for your valuable suggestion.

R4:

Thank you for your comment. The poisoning expert’s core role is to leverage the complementary set concept to facilitate routing, rather than to contribute to output during inference. Please review the process details in R1. We deliberately mask the poisoning expert during inference because it acquires noise-related bias during Stage I, so its outputs inherently carry noise-related bias. If not masked, the model will be affected by the incorrect knowledge learned from noise, which in turn reduces the model's performance. This phenomenon is validated in Section 5.2 of the manuscript (Table 4 and lines 330).

We also believe that LLMs during training typically rely on large-scale datasets of varying quality, inevitably containing noise, whereas the testing phase usually employs high-quality data for specific tasks. This fact is well supported by “Learning from noisy labels with deep neural networks: A survey.” We will include this reference in the camera-ready version. Thanks again!

R5:

We clarify our statement in line 42 as follows in our paper. The term “noise type” refers to the scenario-specific noise patterns and characteristics of each dataset in the context of its corresponding task, rather than to a classification of noise as discrete or continuous.

In contrast, LoPE is not tailored to any specific downstream task or domain, and can thus be widely applied to the parameter-efficient fine-tuning of large language models. In our main results, LoPE has already been validated on over 60 diverse NLP tasks, demonstrating its broad applicability. Its design does not rely on any fixed or pre-defined noise type; instead, it incorporates noise at the character, token, label, and structural levels—covering common real-world perturbations (Section 3.2.1).

Additionally, we have included a dedicated comparison experiment against a task-specific denoising model to further evaluate LoPE’s generality in downstream applications. We have conducted an investigation of the representative denoising method LeCoRE (specifically designed for conversational search tasks). We followed their evaluation protocol and conducted experiments under comparable settings. The experimental results are presented below:

R6:

Thank you for your valuable comment! Lope is a general fine-tuning architecture widely applicable to various noise environments and data types. We consider that it does not conflict with other denoising methods and can be widely combined with them. Therefore, we primarily compared it with other PEFT methods of the same category.

Despite the above discussion, we fully agree with your point and have conducted an investigation of the two representative denoising methods. The experimental results are presented in R5. We will include more comparative experimental results and corresponding analyses in the camera-ready version.

R7:

Thank you for your comment. Regarding the phrase “generate output with relatively clean knowledge” in the caption of Figure 2, although we hope that the model can produce clean outputs, in practice, it is difficult for the model to consistently generate perfectly clean and ideal results, which is why we avoided using absolute terms. Nonetheless, we sincerely appreciate your careful review and will refine this expression in the camera-ready version. We will also conduct a thorough review of the entire paper to ensure clarity throughout. Thank you again!

For all typos, grammatical errors, and unclear notations, we will carefully revise them one by one. Despite these writing issues, we sincerely argue for the value of this work. As mentioned earlier, we have completely eliminated the need for manual noise cleaning by adopting the opposite approach of noise injection. Moreover, we believe the proposed poisoning expert mechanism offers broad inspiration for diverse scenario tasks in the future. Based on the feedback from all the other reviewers, our paper is well-worked and of great value to the academic community. Therefore, we warmly welcome continued discussion and exchange, and we sincerely hope you can take the comprehensive value of this work into consideration. Thanks again!

Thanks for your time and feedback! We admire your deep understanding of this work and your extremely valuable comments. The core of our work that you considered is absolutely correct, and all other reviewers agree with you. Below, we summarize and explain your concerns:

1. This paper mostly compares against other PEFT methods like LoRA and HydraLoRA, but I'm missing comparisons with actual denoising methods (LeCoRE, MICL, etc.). I'd like to see them pit their method against mainstream denoising techniques - that would make their case much stronger.

Q1 Response: Thank you for your valuable comment! In general, aside from the performance improvement, our main advantage is that we completely eliminate the need for data cleaning. In contrast, by adding noise at a low cost, we achieve the goal of "Noise Robustness Through Noise." This not only overcomes issues like uneven data distribution caused by data cleaning but also provides broad inspiration for diverse scenario tasks that are challenging to model or solve directly. Since it is a general fine-tuning architecture widely applicable to various noise environments and data types, we consider that it does not conflict with other denoising methods and can be widely combined with them. Therefore, we primarily compared it with other PEFT methods of the same category.

Despite the above discussion, we fully agree with your point and have conducted an investigation of the two representative denoising methods you recommended: LeCoRE and MICL. For LeCoRE, a method specifically designed for conversational search tasks, we followed their evaluation protocol and conducted experiments under comparable settings. The experimental results are presented below:

The experimental results demonstrate that our method achieves superior performance across all evaluation metrics, indicating the generalizability of LoPE to domain-specific tasks. As MICL is a multi-modal model that presents challenges for direct comparison and the author does not have open-source code, we have not included its results at this stage. We commit to conducting comprehensive and scientifically rigorous evaluations of various denoising models across different scenarios, including LeCoRE and MICL, and will present the corresponding results and analyses in the camera-ready version. Thanks again!

2. This paper primarily tests with 5% noise injection, but then cite research saying real-world noise can be 8%-38.5%.

Q2 Response: Thanks for your comment! As mentioned in Chapter 4.2 of our original text, given that the original dataset already contains some inherent noise, we used an additional rate of 5% to create a more challenging environment, where they work together with the inherent noise in the dataset and collectively affect the model.

2.1 Figure 4 shows different noise ratios but doesn't dig into why performance drops at higher levels.

Q2.1 Response: Due to space limitations, we provided only a brief discussion of this phenomenon. Here, we present a more comprehensive analysis of the observed performance drop under high levels of HyNoIse.

Our analysis identifies several key contributing factors:

(1) Negative Effects of Excessive Noise: When the HyNoIse ratio exceeds 10%, we observe a significant drop in performance. This is because excessive noise impairs the router’s ability to distinguish between clean and noisy data, leading to confusion in expert weight allocation and ultimately weakening the model's discriminative capacity to a certain extent.

(2) Distortion of Data Distribution: As the HyNoIse ratio increases, the statistical properties of noisy samples can overwhelm and potentially disrupt the underlying distribution patterns of the original data.

(3) Randomization of Weight Allocation: As the HyNoIse level rises, the embedding vectors of noisy samples increasingly overlap with those of clean data. This expanded overlap prevents the router from effectively separating the “clean” and “noisy” paths, resulting in an almost uniform distribution of expert weights. Consequently, the originally designated "poisoning expert" (BD) is forced to process clean data, while the normal experts (e.g., B1/B2) are exposed to noisy samples, undermining their role as independent repositories of "clean knowledge."

To aid understanding, we offer an analogy (imperfect, but conceptually helpful):

Low HyNoIse: Similar to giving the model a “vaccine” — a small dose of virus (noise) that stimulates immunity (robustness).

High HyNoIse: Comparable to injecting a large amount of virus directly, overwhelming the immune system (model) and compromising normal functionality.

2.2 Would love to see how this holds up at 10%, 20%, 30% noise - does it collapse or gracefully degrade?

Q2.2 Response: Thank you for your interest! We have conducted extensive supplementary experiments in this regard and discovered some interesting phenomena. We systematically evaluated noise ratios of 3.5%, 5%, 8%, 10%, 20%, and 30% in the noisy data. Partial results are shown below.

From the results, it is evident that while overall performance does decline with increasing noise levels, it does not completely collapse. We analyze the following reasons:

(1) Our method delays performance collapse under higher noise conditions.

(2) The overall quality of fine-tuning datasets impacts final inference performance. When the noise reaches a certain level, it blurs the boundary between clean data and noise, and fine-tuning becomes harmful, highlighting the importance of fine-tuning data quality.

(3) When data noise rises to a certain degree, the accuracy decline rate for mathematics datasets requiring high precision actually slows down. This may be because mathematical pattern-related content in the fine-tuning dataset has already been corrupted, resulting in less impact on the model itself (the model may no longer recognize that it is learning mathematics-related knowledge).

In the camera-ready version, we will present the complete set of these supplementary experiments to provide a comprehensive view of our work and enrich the research community's understanding of LLM data noise. We will include comprehensive results across various noise ratio settings in the camera-ready version, along with an in-depth analysis of the observed performance variations and the underlying differences between real-world noise and synthetically injected training noise in the corresponding section.

3. It would be better that include visualization or analysis of its internal representations to describe what exactly is this poisoning expert learning.

Q3 Response: Thank you for your valuable comment! Following your advice, we have conducted a comprehensive visualization of the A and B parameters in LoPE, which follows the LoRA matrix visualization method released in the HydraLoRA GitHub repository. Although we cannot provide visualizations here due to the format limitation of the rebuttal, we list the key data and conclusions here. Overall, our findings show that during the entire two-stage fine-tuning process, the position of matrix A in the visualization space changes very little, which is fully consistent with our previous conclusions in the manuscript, and also aligns with the HydraLoRA paper’s conclusion that matrix A represents general knowledge. In contrast, we observed significant positional changes between the standard expert B matrix and the poisoning B matrix across the training stage. We attribute this phenomenon to the B matrix capturing knowledge contaminated by noise, which is fundamentally different from the clean knowledge encoded by other experts. This aligns with our hypothesis that the poisoning expert effectively diverts tokens affected by noise. We will add the complete visualization process and content mentioned above in the camera-ready version and provide corresponding theoretical analysis and explanations. Thank you again for your insightful suggestion!

Thank you again for taking the time to review our response. We sincerely hope that our response can solve your concerns and better evaluate the comprehensive value of our work. Thanks again!