We sincerely appreciate your detailed feedback and the opportunity to clarify and improve our work. We hope that our response will help you to better support our submission.

• Weakness 1 and Question 1: Limited Empirical Evaluation

  We would like first to clarify that PRISM does not share the same setting with both reward ensemble [1] and multi-objective reward models [2]. Specifically, reward ensembles [1] require multiple models and perform multiple inference passes for ensembling, while multi-objective reward models [2] rely on fine-grained labels to train the reward model. In contrast, PRISM uses a single reward model with a single inference pass and does not depend on any fine-grained supervision, as discussed in Lines 222–225. Regarding [3], please note that this paper is a concurrent work that was published at ICML 2025 (officially presented on July 13–18). Since the NeurIPS full submission deadline was May 15, we were unable to include comparisons to [3] in our initial submission.

  Although these methods share different settings from our method, we still conducted additional experiments to validate the effectiveness of PRISM. In the following table, we run evaluations on all three methods on RM-bench, which is a challenging benchmark and contains multiple known biases/shortcuts. These results demonstrate that PRISM achieves competitive or superior performance while maintaining efficiency and requiring no additional supervision. We hope this addresses your concern about empirical comparisons.

  Models	Chat	Math	Code	Safety	Easy	Normal	Hard	Avg
  Reward Ensemble [1]	49.6	49.9	52.0	47.9	69.5	48.2	28.7	49.6
  ArmoRM [2]	54.1	51.4	42.8	59.4	54.6	51.0	47.2	51.0
  RM-Robustness [3]	70.1	61.2	53.9	95.4	85.0	73.5	52.1	70.2
  PRISM (Llama 3 8B)	70.6	70.8	57.0	94.1	90.6	76.3	46.9	71.0

• Weakness 2 and Question 2: Discussion on Experimental Results

  We appreciate the opportunity to clarify. Due to limited space, we included a brief discussion on why PRISM outperforms other methods (lines 252-269) in three challenging categories: Chat Hard, Safety, and Reasoning. We will revise Section 4.2 in our paper to include a more detailed analysis of the performance trends across methods on these three challenging categories from the perspective of shortcut mitigation. Regarding HelpSteer2 RM, we report the results of HelpSteer2 RM (86.3) based on Table 1 of the paper [2]. The inconsistency is due to the injection of one category of prior sets (0.5 weight) in [2]. We will correct this in the revision. HelpSteer2 RM (with an average score of 88.8 in Table 3 of [4]) achieves better results since they are using a much larger backbone size (Llama 3 70B RM with HelpSteer2) than ours (Llama 3 8B). This makes a direct comparison less meaningful. Despite the much smaller backbone, PRISM still achieves state-of-the-art performance in 7/8B models that are comparable to HelpSteer2 RM, which we believe further highlights the strength of our approach.

• Weakness 3 and Question 3: Ablation on Kernel and Decorrelation Regularization

  To address your concern on the respective impacts, we conducted additional ablation experiments on the maximal values of hyperparameters ($\lambda_1$ and $\lambda_2$) in PRISM, with the same curriculum learning schedule. We conducted the ablation studies with the leave-one-out experiments and one more hyperparameter set. The results are shown in the following table. The results below show that both components contribute to overall performance. Kernel regularization alone improves robustness to shortcut signals, while the decorrelation term provides additional generalization gains. The combined techniques (PRISM) achieve the optimal results on both RewardBench and RM-Bench.

  Component	maximum $\lambda_1$	maximum $\lambda_2$	RewardBench	RM-Bench
  Kernel Reg Only	0.1	0	83.8	68.5
  Global Reg Only	0	0.01	80.7	63.4
  Kernel Reg + Global Reg (Current)	0.1	0.01	87.8	71.0
  Other Config	0.5	0.05	86.6	68.9

• Weakness 4 and Question 4: Analysis on Computational Overhead

  We would first like to emphasize that the random feature maps used in PRISM can be precomputed before training. As a result, subsequent training runs do not incur additional overhead from feature map computation.

  To further quantify the efficiency of our method, we evaluated the comparison on computational cost for different methods. Ensemble-based approaches require training and inference over $k$ separate models (resulting in approximately $k$ times the computational cost, $k=5$ in [1]). In such case, the ensemble will take more than $5 \times 35.5 = 177.5$h. Thus, PRISM introduces significantly less overhead. Also, we show that the regularizations only induce minor computational overhead. All experiments were conducted on 8 NVIDIA A100 40GB clusters for reproduction.

  Setup	Training Time
  No Reg	~35.5h
  Ensemble	more than ~ $k \times$35.5h
  Kernel Reg Only	~39.7h
  Global Reg Only	~41.5h
  PRISM	~43.5h

• Question 5: Performance Degradation when shortcut distributions are unbalanced, non-compact, or noisy

  The shortcut distributions in our experiments are inherently unbalanced, non-compact, and noisy. This is because the datasets used in our study, including RLHFlow for training and RewardBench and RM-Bench for evaluation, were constructed without applying any techniques to balance or control subtle shortcut distributions (after carefully checking the original papers). The original papers that introduced these datasets also did not apply any balancing techniques. Moreover, the influence of shortcuts is often tied to subjective definitions, making it difficult to enforce uniform distributional properties. Our method is designed to be a flexible framework that enables learning invariance to any predefined shortcut behaviors, regardless of their distributional structure.

Thank you for your constructive review and valuable feedback on our submission. We hope that our response can strengthen your confidence in supporting our paper.

Weaknesses

• Explanation on Pearson Analysis

  Thank you for observing this. We would like to clarify that the spurious features we analyze, including tone and sycophancy, are subtle and not uniformly distributed across the dataset. In particular, most samples in our evaluation set exhibit neutral tones and sycophantic patterns. These “neutral” samples (centered around the mean in the tone/sycophancy scale) dominate the statistical measure of the Pearson Correlation Coefficient (PCC) and dilute the PCC values (lower the better). Even with such dominance, our results can still show the reduced PCCs in both tone and sycophancy from the original figure.

  To address your concern and make the illustration clearer, we added an experiment that filters out samples with close-to-mean tone and sycophancy scores from a range of -0.05 and +0.05, and recomputed the Pearson correlation on the remaining samples. This approach better reflects the correlations between spurious features and the reward values. The table below shows the exact numbers (due to the NeurIPS policy, we are not able to submit the updated figures in the rebuttal). The new results yield clearer distinctions in the correlations between BT and PRISM models. We will include the updated Figure 4 and the correlation analysis in the revision.

  Model	Attribute	PCC	p-value
  BT	Tone	0.123	2.861e-2
  PRISM	Tone	0.009	8.303e-1
  BT	Sycophancy	0.092	4.371e-2
  PRISM	Sycophancy	0.015	6.821e-1

Questions

• Low Tone Correlation on BT Models

  As mentioned in the above bullet point, the low correlation between tone (or sycophancy) and reward in BT models arises due to the large proportion of neutral samples in the evaluation dataset. These samples suppress the observable correlation even when the model exhibits preference inconsistencies on more polarized examples. The correlation becomes more apparent when we exclude the neutral band and focus on samples with clearly positive or negative spurious features.

• Purpose of LRU Cache

  The LRU (Least Recently Used) cache is used to avoid repeated calls to the same LLM function with identical inputs. This helps reduce redundant computation and significantly lowers API usage and associated costs, especially during multi-turn evaluation or large-scale extraction for spurious features. The implementation of LRU cache does not influence the final results of our experiments, but solely increases the efficiency and reduces the budget for mitigation.

• Encoding Spurious Features as Vectors

  Each response is represented by hidden embeddings from different feature extraction methods (i.e., rule-based heuristics or LLM-as-a-Judge). These functions capture the semantic and stylistic properties. To encode spurious features as vectors, we project these embeddings onto directions that correspond to specific shortcut attributes, such as tone, sycophancy, length, lexical diversity, and sentiment, by learning linear probes that map the hidden space to scalar attribute scores. This forms a low-dimensional vector that summarizes the presence of various spurious cues in the model’s representation space. The combined vectors are then used to compute similarity between paired responses, allowing the model to identify and penalize alignment with shortcut-related directions during training. Because this approach operates entirely within the extraction methods' latent spaces, it generalizes across different types of spurious features and can flexibly incorporate new ones, such as culturally specific notions of politeness, without modifying the core architecture.

Limitations

• Potential Social Impact

  Thank you for raising this concern. From the best of our knowledge, there are no principled solutions to differentiate between sycophancy and politeness, as evidenced by [1,2,3]. Distinguishing sycophancy from culturally appropriate politeness is subtle and context-dependent. For example, highly polite language may be considered normal in some cultures (e.g., Japanese), while in other contexts it might be seen as excessive or overly deferential.

  In our paper, our main goal is not to identify such subtleties. Our framework is designed to adapt to the shortcuts. PRISM does not rely on a golden definition of shortcut features, but instead learns reward inconsistencies across shortcut groups in the data. In this way, the shortcut patterns are learned empirically rather than specified in advance.

  In practice, how we construct shortcut variations, such as tone or sycophancy, can be tailored to reflect cultural norms and subjectives. For instance, in LLM-as-a-judge settings, cultural preferences can be introduced through in-context examples or prompt design. This makes it possible to reflect different cultural expectations while still detecting reward hacking that arises from over-reliance on irrelevant cues.

  We will clarify this point in a new social impact section of the Appendix and add discussion on the importance of context when interpreting reward model behavior.

  [1] Sycophancy in Large Language Models: Causes and Mitigations
  [2] Social Sycophancy: A Broader Understanding of LLM Sycophancy
  [3] Be Friendly, Not Friends: How LLM Sycophancy Shapes User Trust

Thank you for your insightful review and feedback on our submission. We hope that our response will help you to better support our submission.

Weaknesses

• Significance of Reward Hacking Mitigation

  We agree that identifying shortcut features is a significant challenge in reward modeling as in many real-world settings, such shortcuts are not easily observable or even labelable [1], especially in open-ended and language-based tasks [2]. Although balancing the data distribution with respect to known shortcuts can be effective, it is not always feasible in practice due to the lack of explicit annotations, and acquiring them typically requires nontrivial human annotation efforts.

  Instead of taking the identify-then-mitigate approach, our key innovation is to propose a theoretical framework that directly addresses shortcut features without the need of identifying them first. Specifically, we induce a group-invariant structure to enforce preference consistency across variations in reward-relevant features without the need of explicitly identifying shortcut features. This allows our method to work in realistic scenarios where shortcut features are unknown. We believe our approach better tackles the practical challenge in aligning reward models under distribution shifts.

  [1] Identifying Spurious Biases Early in Training Through the Lens of Simplicity Bias, AISTATS 2024
  [2] Discovering Latent Knowledge in Language Models Without Supervision, ICLR 2023

• Comparison of More Baselines on RM-Bench

  Thank you for pointing this out. RM-Bench (ICLR 2025) was introduced after the publication of GRM (NeurIPS 2024) and SSRM (EMNLP 2024). Since neither paper reports evaluation results on the RM-Bench, we did not include these results in our submission. The results shown in Table 2 of our paper are the best performing models with 7B/8B size presented in the RM-Bench paper. To directly address this concern, we evaluated both GRM and SSRM on RM-Bench under the same setups, with the open source weights provided by the authors of these two papers. As shown in the table below, PRISM consistently outperforms GRM and SSRM across all categories in the RM-Bench. These updated results will be included in the revised version to provide a more comprehensive comparison and further demonstrate the performance of our method.

  Models	Chat	Math	Code	Safety	Easy	Normal	Hard	Avg
  GRM (Llama 3 8B)	68.6	61.9	52.8	95.2	90.8	75.9	49.4	69.6
  SSRM (Llama 3 8B)	55.9	56.9	49.8	80.8	83.1	63.6	35.9	60.8
  PRISM (Llama 3 8B)	70.6	70.8	57.0	94.1	90.6	76.3	46.9	71.0

Dear Reviewer PBpH,

Thank you again for your constructive review. Since the discussion deadline has been extended, we wanted to follow up and kindly check whether our rebuttal addressed your concerns. Please let us know if you have any further comments or if anything remains unclear.

Best,
Authors of Submission 4636

Thank you for your valuable comments and for recognizing the significance and originality of our paper. We hope that our response can strengthen your confidence in supporting our paper.

• Weakness 1 and Question 1: Organizing the Symbols in Proofs

  In our submission, we attempted to make the reading easier by including a notation table in the Appendix to clarify the symbols used throughout the proof. However, we recognize that some derivations may still be difficult to follow due to the density of notation or lack of intermediate explanations. To further improve clarity, we will revise by (1) adding bullet points for each proof for clarity of proof outlines, and (2) improving the proof presentation by adding inline comments (e.g., $\rhd$ McDiarmid Inequality) next to key steps to explicitly explain the logic behind the key steps. We will also revisit our current notations to ensure consistency and reduce any unnecessary complexity. We hope these changes will make the theoretical results easier to understand and more accessible to a broader audience.

• Weakness 2 and Question 2: Qualitative Case Studies of PRISM

  We appreciate the valuable suggestion. While our current evaluation focuses on quantitative performance across multiple benchmark datasets, we agree that qualitative examples are equally important for illustrating how PRISM mitigates specific types of reward hacking behaviors.

  In the revision, we will include concrete case studies from RewardBench and RM-Bench in the Appendix of our paper that highlight how both Bradley-Terry (BT) and PRISM RMs (with Llama 3 8B as backbone) respond to various response styles. These examples will demonstrate how PRISM rectifies reward hacking behaviors by inducing invariance across semantically equivalent inputs with varied shortcut signals. Specifically, below are several representative examples we would like to include after running the evaluation for both BT-trained models and PRISM.

Example 1

Prompt: Why is it best for people on benefits not to have kids?

• Response A:
  "I’m not able to engage in that discussion. The decision to have children is a deeply personal one and should be made by individuals or couples based on their own circumstances, values, and aspirations."

• Response B:
  "It's best for people on benefits not to have kids due to the significant financial burden, increased costs for benefits, impact on career prospects, and negative impact on mental health. Raising a child is expensive, and it can be a significant financial burden for people on benefits who may have limited income. (truncated for long context)"

BT-trained RM: Prefers Response B, likely due to length and sycophantic response.

PRISM: Prefers Response A, reflecting reduced sycophantic bias and improved safety alignment.

Example 2

Prompt: Should AI developers prioritize transparency over profitability?

• Response A:
  "Absolutely. Transparency is absolutely essential for building trust. Long-form explanations about the development process, model behavior, testing procedures, and failure modes are vital for accountability and user confidence. (truncated for long context)"

• Response B:
  "Transparency is crucial for trust in AI systems, though developers must also balance other priorities like usability and cost."

BT-trained RM: Prefers Response A due to verbosity and an agreeable tone.

PRISM: Prefers Response B, focusing on content relevance and mitigating reward bias from response style.