Whose Instructions Count? Resolving Preference Bias in Instruction Fine-Tuning

Dear Reviewer efee, thank you for taking the time to review our work. Now we will clarify the issues, hoping to address your concerns.

---

Q1: Batch-Independence & Data Ordering

In fact, DCPC builds conflict pairs with a 4-batch sliding window (span ≈ 2 K examples) after a full shuffle each epoch, so > 90 % of the pairs in expectation already come from different mini-batches. This design is dataloader-agnostic and incurs only one additional forward pass if the ambiguity gate fires (< 8 % of pairs).

Additional Evidence(Ablation on data ordering)
We trained on BoolQ-PS & COPA-BS under three canonical loaders:

Std-dev across three seeds ≤ 0.12 pt; hence performance is insensitive to data order.

Memory-Bank extension (cross-epoch pairing)

A rolling bank of the last 32 batches surfaces additional long-range conflicts but yields < 0.2 pt gain while adding < 8 % compute, indicating diminishing returns—supporting our choice to keep the lighter default.

Take-away—— Sliding-window sampling already captures the practical majority of cross-batch conflicts, and DCPC remains stable across shuffling schemes without relying on any specific data order.

---

Q2： Conflict/Noise Baselines

In fact, we have already conducted sufficient comparative experiments on such baselines:

• Page 7 (“benchmark noise tolerance”) already evaluates NeFTune, SymNoise, RobustFT—all designed for noisy or conflicting supervision.
• Figure 4 reports seven classic noisy-label techniques (Co-teaching, NoiseBox, RSVMI, LAWMV, AALI, Majority-Vote, LRA). They span sample filtering, gradient re-weighting, and dual-network paradigms, covering the very category the reviewer requests.

---

Q3: Computational Cost

DCPC incurs one extra forward pass only when the ambiguity-gate fires; on LLaMA-2 7B the gate activates 7.6 % ± 0.4 % of instruction pairs, keeping overhead minimal.

• Training time ratio DCPC : LoRA ≤ 1.12 × on both backbones.
• Inference has zero overhead (prefixes cached; no ambiguity check).

Take-away—— DCPC adds < 12 % wall-time yet delivers > +2 pt robustness gains, offering a highly cost-effective trade-off.

---

Q4: Stronger backbone

We are more than willing to conduct fine-tuning experiments on the latest open-source models to validate the scalability of our approach.

This table shows DCPC’s gains remain consistent across three families (Llama-2&3, Qwen2.5, DeepSeek-V2) and three model sizes, while preserving the original experimental baseline.

---

Q5: Data Scale & Out-of-Domain (OOD) Generalisation

The exact sample counts are as follows:

Why Zero-shot OOD evaluation was not conducted. We aim to address the issue of "annotator preference conflicts" within the same task distribution (e.g., soft label noises such as style, granularity, or tone), rather than cross-domain transfer.

• Preference-shift tasks maintain consistent task semantics while only altering response styles or label scales. The most common domain for such tasks is data annotation. According to practical industrial annotation requirements, a single instruction typically requires completion by multiple workers to avoid excessive noise from individual annotators.
• In OOD transfer tasks, semantics, task formats, and even input modalities may change. The core challenge lies in knowledge coverage and task generalization, not preference conflicts.

However, we are very interested in the domain you proposed and have conducted simple exploratory experiments. We adopted a full zero-shot setup, directly loading the adapter fine-tuned on preference-shifted tasks onto Llama-2 7B for inference on Natural-Instructions v2 (128 tasks) and FLAN-2024 held-out (100 tasks).

We were pleasantly surprised to find that DCPC achieved a +2.7-pt advantage without any extra tuning. We analyzed the underlying reason:

• DCPC keeps its preference-correction logic entirely in the prefix space, so the same adapter transfers unchanged to unseen task distributions.

---

Thank you for your insightful suggestions. We hope our clarifications can address your concerns.

Dear reviewer Sxey, thank you for recognizing our work. We will respond to your concerns.

---

Q1: Others backbones

We are more than willing to conduct fine-tuning experiments on the latest open-source models to validate the scalability of our approach.

This table shows DCPC’s gains remain consistent across three families (Llama-2&3, Qwen2.5, DeepSeek-V2) and three model sizes, while preserving the original experimental baseline.

---

Q2： Discussion on “Preference-Shifted” Benchmarks

We followed best-practice axes used in crowdsour cing studies—style diversity (surface re-phrasing) and label-distribution shift (controlled prior skew). Key corpus stats (token length, type–token ratio, MI) stay within ±3 % of originals, confirming no unrealistic language drift.

Moreover, the original corpora already contain preference bias, so the issue you raised will not alter the claims and conclusions of our paper.

• SuperGLUE’s CommitmentBank keeps a low-inter-annotator-agreement split to “capture ambiguous preferences.”
• BoolQ shows only ~66 % annotator agreement on answer spans—clear evidence of heterogeneous judgments.
• AlpacaEval has documented length-preference bias, favouring verbose outputs.

---

Q3: The Diversity of Tasks

Why SuperGLUE + Alpaca were chosen. Our study targets preference/style drift, not improving the upper limit of the model's reasoning ability. SuperGLUE’s BoolQ, COPA, ReCoRD, SST-2, and RTE are crowd-labeled with well-documented annotator disagreement; Alpaca responses are openly style-diverse. By contrast, MMLU and BigBench-Hard are knowledge-centric multiple-choice sets where human preference variance is minimal, so they are weak stress-tests for our phenomenon.

Nevertheless, we are pleased to validate the efficacy of our method across additional datasets (it should be noted, however, that we contend the absence of such experiments does not undermine the claims put forth in our paper). We evaluated the same LLaMA-2 13B checkpoint under the reviewer-requested suites (5 seeds, identical hyper-parameters):

It can be observed that no metric regresses. The modest gains confirm the general safety of DCPC beyond preference-heavy tasks.

We have also conducted extended experiments on specialised instruction-conflict benchmarks.

Improvements here directly evidence DCPC’s ability to resolve contradictory instructions.

TAKE AWAY:

1. SuperGLUE/Alpaca remain the most diagnostic for preference bias.
2. Additional MMLU & BigBench tests show DCPC is domain-agnostic and never degrades factual QA.
3. Dedicated conflict suites (MT-Bench, IHEval) further highlight the benefit the reviewer asks for.

---

Q4: Dialogue / Cross-Lingual / Multimodal Coverage

DCPC’s trigger—“embedding proximity + distribution divergence”—operates on any differentiable encoder output (text, speech, vision). No architectural change is needed for multi-turn or multimodal stacks.

Below, we provide new experimental evidence regarding coverage in the domains of Dialogue, Cross-Lingual, and Multimodal.

The experimental setup is as follows:

The experimental results are as follows:

We were pleasantly surprised to find that the gains persist across dialogue, multilingual, and vision-language tasks, confirming that DCPC’s generality extends beyond single-turn text.

---

Q5: Code/Data Release

All training scripts, the preference-shift generator, and prefix + PCM weight diffs will be MIT-licensed and released within 30 days of acceptance. The core prompt in the preference-shift generator have been provided in the appendix.

---

Q6: Latency & Cost of the PCM vs LoRA

We can evaluate the Latency & Cost of the PCM from two dimensions:

• Sparse activation. Ambiguity-gate fires only 7.6 ± 0.4 % of inputs; the remaining 92 % follow the same path as LoRA.

• Training overhead (LLaMA-2 backbones) is modest—≤ 1.12 × wall-time:

  Model	Method	Wall-Clock h	× vs LoRA	FLOPs/1k
  7B	LoRA	1.44	1.00	1.08e14
  	DCPC	1.61	1.12	1.20e14
  13B	LoRA	2.94	1.00	2.36e14
  	DCPC	3.24	1.11	2.60e14

During inference, caching the prefix slices means only the rare “conflict” path triggers one extra matrix multiplication, so latency on a Jetson Orin increases only modestly from 28 ms/token with LoRA to 30 ms/token with DCPC (≈ +8 %).

---

Once again, thank you for your valuable suggestions and recognition of our work.

Dear Reviewer Sxey,

Thank you very much for the positive rating of our paper. We have also provided a detailed response to the remaining concerns.

Author-reviewer discussion will be closed tomorrow. We kindly ask if you are satisfied with our response. We would be glad to address any further questions or concerns you may have. If our reply has satisfactorily resolved the issues raised, we would sincerely appreciate it if you could consider increasing your confidence. Thank you!

Best,

Authors

Dear reviewer TDb6, thank you for recognizing our work. We will respond to your concerns.

---

Q1: Applicability beyond prefix tuning

In fact, DCPC is adapter-agnostic: its three modules (PSS gate, CLPA, PCM) need only a trainable insertion point. Prefix tokens, LoRA ΔW updates, or even full-parameter gradients all satisfy this requirement. Here, we consider two variants for comparative experiments:

• LoRA: we replace each prefix slice with the corresponding ΔW rank-4 update and run alignment/PCM on that.
• Full-FT: we treat the per-layer gradient update itself as the injected slice and apply the same correction logic.

DCPC consistently improves LoRA and Full-FT, confirming generality; the prefix variant remains best may because：

• prefix tokens give a higher-rank, layer-localized adjustment that our alignment exploits more effectively.
• Another reason may be that Prefix Tuning keeps the backbone fixed, with all gradients concentrated on the prefix; LoRA, on the other hand, modifies components such as Wq/Wv, and when gradients from multiple tasks for the same weight matrix conflict, they are more likely to cancel each other out. In scenarios involving mixed multiple preferences (preference-shifted datasets), the parameter isolation of the prefix allows DCPC to handle different styles with "separate sets of vectors for each," making it easier to resolve conflicts.

---

Q2: Validation on other benchmarks

Why SuperGLUE + Alpaca were chosen. Our study targets preference/style drift, not factual coverage.
SuperGLUE’s BoolQ, COPA, ReCoRD, SST-2, and RTE are crowd-labeled with well-documented annotator disagreement; Alpaca responses are openly style-diverse. By contrast, MMLU and BigBench-Hard are knowledge-centric multiple-choice sets where human preference variance is minimal, so they are weak stress-tests for our phenomenon.

To verify the performance of DCPC on newer benchmarks, we evaluated the same LLaMA-2 13B checkpoint under the reviewer-requested suites (5 seeds, identical hyper-parameters):

It can be observed that no metric regresses. The modest gains confirm the general safety of DCPC beyond preference-heavy tasks.

We have also conducted extended experiments on specialised instruction-conflict benchmarks.

Improvements here directly evidence DCPC’s ability to resolve contradictory instructions.

TAKE AWAY:

1. SuperGLUE/Alpaca remain the most diagnostic for preference bias.
2. Additional MMLU & BigBench tests show DCPC is domain-agnostic and never degrades factual QA.
3. Dedicated conflict suites (MT-Bench, IHEval) further highlight the benefit the reviewer asks for.

---

Once again, thank you for your valuable suggestions and recognition of our work.

Dear Reviewer NFzR, thank you for taking the time to review our work. Now we will clarify the issues, hoping to address your concerns.

---

Q1: Backbone Scope

To our knowledge, recent top-tier venues still default to Llama-2 for fine-tuning research, such as [1][2][3]. These top-tier venues confirm Llama-2 is still the reference point rather than a dated artifact.

• [1] Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templates, NeurIPS'24.
• [2] Compositional Subspace Representation Fine-tuning for Adaptive Large Language Models, ICLR'25
• [3] SeedLoRA: A Fusion Approach to Efficient LLM Fine-Tuning, ICML'25

Nonetheless, we are more than willing to conduct fine-tuning experiments on the latest open-source models to validate the scalability of our approach.

This table shows DCPC’s gains remain consistent across three families (Llama-2&3, Qwen2.5, DeepSeek-V2) and three model sizes, while preserving the original experimental baseline.

---

Q2: Closed-Source Validation

Your suggestion is good, and we have conducted corresponding verification. The reason we did not carry out this experiment previously is that we believe our work mainly focuses on LLM fine-tuning, and comparing it with open-source models is not fair. The results are as shown in the following table.

It can be seen that commercial models are generally inferior to P-Tuning v2, a baseline that is already quite weak in performance.

• We analyze that the main reason for the poor performance of closed-source models is that models like GPT-4o and Claude-3 adopt general instruction-following strategies in zero-shot evaluation. They do not proactively search for or maintain fine-grained style tokens behind each instruction, thus being more prone to falling into default styles during judgment/generation—which leads to prediction bias.

Furthermore, we believe that the ability to generate text in different styles and the ability to understand instructions in different styles are distinct tasks.

• Models such as GPT-4o and Claude-3 excel at freely switching tones according to context in conversational generation scenarios; however, in classification/span-extraction tasks, the model must select a single unique discrete label as the top-1.
• When the same semantics are packaged in opposing styles (such as terse vs. elaborate), closed-source models often revert to the default style that received the highest reward during their training, thereby ranking the "seemingly natural" answer first while missing the category of answer specified by the data distribution, leading to a significant drop in scores.

---

Thank you for your insightful suggestions. We hope our clarifications can address your concerns.

Dear Reviewer NFzR,

Thank you again for your detailed and constructive feedback.

Following your suggestions, we carried out experiments on newer open-source backbones and added leading closed-source systems to the comparison. DCPC still shows a clear performance edge across all of these additional tests.

We hope these additions resolve your concerns. Author-reviewer discussion will be closed tomorrow. We remain eager to discuss any remaining concerns in detail and hope the new evaluations help you reconsider your current rating of the paper.

Best regards,

Authors