We thank the reviewer for their feedback. Below, we address your concerns regarding KLDO’s empirical performance, the robustness score, model selection, and emphasize the theoretical scope of our work.

Summary: Purpose and Contribution of Our Work

This paper is primarily theoretical, introducing a unifying framework that interprets LLM alignment as a process of divergence estimation. We show that popular alignment methods—such as KTO, BCO, and DPO—implicitly estimate specific divergences (e.g., total variation, Jensen-Shannon), which in turn naturally induce latent separation between safe and harmful prompts. As a concrete instantiation, we propose KLDO, an alignment method based on KL divergence, and extend this to a broader family of FDO methods that estimate arbitrary $f$-divergences. We prove that divergence-based alignment satisfies an alignment consistency property, which guarantees the emergence of separation—especially when trained with compliance-refusal data. Our experiments, conducted under controlled settings across diverse open-source models, validate these theoretical predictions and show a strong correlation between separation and robustness. The goal of these experiments is not to achieve state-of-the-art performance, but to isolate and demonstrate the theoretical mechanisms underlying alignment. KLDO serves as a proof-of-concept, and achieves competitve and consistent performance across robustness and utility. Our framework provides theoretical guarantees for any FDO aligner and opens the door to future exploration of alternative divergences, such as Rényi or custom-designed measures, essentially a plug-n-play mechanism to generate arbritrary alignment optimizers.

Robustness Score: Clarification

To compute the overall robustness score, we normalize each benchmark result to the $[0, 1]$ range across all methods for a given model; this helps us reconcile scale differences across different benchmarks.

For metrics where higher is better (e.g., ToxiGen accuracy): $\hat{x}_i = \frac{x_i - \min(x)}{\max(x) - \min(x)}$ ​For metrics where lower is better (e.g., ASR): $\hat{x}_i = \frac{\max(x) - x_i}{\max(x) - \min(x)}$ The final robustness score is the average of these normalized values across all benchmarks. This approach rewards methods that are consistently and relatively strong across various aspects of robustness.

KLDO gets a high score as it is consistently at the top or loses by small margin, than outlier wins on individual tasks.

KLDO vs. BCO: Empirical Justification

KLDO is proposed not simply as another alignment baseline, but as a concrete realization of our divergence-based theoretical framework. Its goal is to demonstrate that divergence estimation provides a principled and generalizable mechanism for inducing separation and improving robustness.

Figure 3 and Table 2 are consistent. If we look at the separation value for qwen2.5 BCO does beat KLDO by a slight margin in that scenario. However, purpose of Figure 3 was to illustrate that all alignment methods naturally induce separation as predicted by our theory from base model. Separation is a bi-product of alignment, which is one of our major contributions (Theorem 4.5) as well. Moreover, in terms of separation we are modest, and we can see KLDO is close second to BCO or beats BCO with huge margin for example gemma2. We did provide plots for all models in section C.6 Latent Space Visualization and it is evident qualitatilvely that KLDO is close to BCO or beats it by a lot like in gemma2. KLDO is consistent.

While BCO may lead slightly in a few metrics, KLDO offers more consistent and balanced robustness, which reflects in the overall score. Moreover, KLDO shows consistent performance in utility benchmarks, while other methods can show significant performance dips for certain configurations (eg. Alpaca eval or GSM: Archangel Gemma BCO performance).

On High ASR for SALAD

SALAD [4] is an exceptionally challenging benchmark composed of attack-enhanced prompts generated by methods such as TAP, AutoDAN, GCG, GPTFuzz, and others. Even commercially aligned models of similar or larger scale struggle on this benchmark. This is why the SALAD leaderboard reports defense scores (i.e., $100 - \text{ASR}$) rather than ASR directly. As shown in Table 5 of [4], our reported ASR values are consistent with expectations for models of similar size. These are some results from the defense against attack enhanced leaderboard for the commercial counterpart or similar to models used in our setting.

Thus, the high ASR values we report should be interpreted in the context of SALAD’s difficulty and standard evaluation conventions—not as unusually poor performance.

Goal was to compare alignment methods in controlled setting not absolute performance

Ultimately, our goal is not to achieve absolute state-of-the-art robustness, but to conduct a relative comparison of alignment strategies under controlled and reproducible conditions.

Model Selection Rationale

We wanted to use a diverse set of open-source LLMs from different organizations (LLaMA, Gemma, Mistral, Qwen) at the 1B–7B scale for (1)To ensure broad representational diversity in model families.(2)To remain within the limits of a university compute budget.

This controlled experimental setup serves to validate our theoretical claims and demonstrate the feasibility of divergence-based alignment. Prior works have indicated alignment inducing separation for various models and sizes. Our work is a deep dive in explaining, justifying the mathematical framework and proving separation is bi-product of alignment.

[4] SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models

We thank the reviewer for their continued engagement and insightful questions. Below we clarify how the theory guides the experiments, the interpretation of robustness metrics, and the broader contributions of the work.

1. Role of Theory and the KLDO vs. BCO Comparison

We believe there is misunderstanding pertaining to the theory-experiment connection and respectfully would like to clarify,

• Figure 2 (Lines 150–160, Section 4.1.1) analyzes the sensitivity of divergences, which is foundational because the alignment objective is built upon the divergence form.
  We show that DPO’s implicit divergence exhibits saturation and lacks convexity (s-shaped) like other divergences, making it insensitive to both small and large distributional changes—undesirable for safety alignment (Essentially, if the underlying divergence itself is bad it caps the upper performance you can expect, it is necessary condition).
  In contrast, KL divergence has unbounded, convex growth, offering a stronger learning signal to penalize distribution mismatch. This insight motivates KLDO as a divergence-guided alignment loss.

• It’s important to note: Such an insight is heuristic and limited (e.g., Fig 2 only considers Gaussian distributions). Moreover, KLDO is not KL divergence itself, but an alignment method derived from a variational form of KL.

• Then, our rigorous result Theorem 4.3 (Lines 201–209) characterizes the alignment consistency of different methods by computing log-probability ratios between aligned and unaligned responses: $\text{DPO} < \text{KTO} < \text{KLDO} \approx \text{BCO}$ Recall our theorem proves that the same $h$ function holds for KLDO and BCO. It means, our theorem doesn't claim the superiority of KLDO over BCO. This ordering pertains to latent separation, assuming convergence. It aligns with the separation scores (Table 2) and visualizations (Appendix C.6). In other word, our theorem doesn't claim the superiority of KLDO over BCO.

In practice, global convergence is rarely achieved, especially in large-scale LLMs. This is where we believe KLDO based on the KL divergence benefits.

• KL divergence's sensitivity gives KLDO sharper gradients, making it more responsive to nuanced distributional differences, especially under limited training budgets.
• As a result, KLDO consistently outperforms DPO and KTO, and shows an edge over BCO in overall robustness (e.g., ASR, ToxiGen) and utility (e.g., AlpacaEval, GSM8K).

Thus, even if KLDO and BCO converge to similar alignment-consistent distributions, KLDO can show an edge in practice, leading to improved robustness and especially utility (which is more sensitive to nuance of distributional changes) in real settings. And given our theorem, we don't expect that KLDO overwhelmingly outperforms BCO.

2. Interpretation of Overall Robustness Score

Our overall robustness metric is computed by normalizing each benchmark to $[0, 1]$ within each model and averaging. Rather than interpreting this as an absolute score, it should be seen as a continuous analog of rank aggregation (i.e., average rank over all benchmark):

• It reflects relative performance across benchmarks.
• It avoids biases introduced by metric scales (e.g., comparing ASR vs. ToxiGen).
• It rewards consistency across robustness axes, which is a critical goal in safety evaluation. Given the diversity of metrics, we believe this is a reasonable and principled aggregation method.

3 Meaningful ASR Reductions in SALAD

Addressing, "ASR reduction from 92% to 89% (e.g., LLaMA2-7B) is meaningful?"

We clarify:

• As discussed in our previous response, SALAD is an extremely difficult benchmark, where even commercial models fine-tuned for safety show defense scores under 20% (Table 5 of [4]).
• Our models are aligned from scratch, without additional safety-specific tuning.

What matters is the relative trend across alignment methods. KLDO consistently outperforms DPO and KTO, and often BCO, across SALAD, ASR, and other benchmarks. Combined with its stable utility, KLDO becomes an attractive choice when safety is a concern.

We agree that methods specifically optimized for safety defenses (e.g., circuit breakers, refusal classifiers) will outperform our results, or any other base model with just alignment losses benchmarks.

However, our goal is different: we aim to understand the intrinsic safety behaviors of alignment methods. Alignment is a foundational step—it does not preclude pre- or post-processing. Our framework clarifies how the divergence used in the alignment loss directly affects robustness and separation. This is not an either-or situation. KLDO or FDO-style methods can be combined with downstream safety techniques. Understanding their inherent tendencies is what this work contributes.

To better appreciate the practical and theoretical significance of our work, we would like to briefly situate it in the evolving history of LLM alignment:

Traditionall alignment method: Reinforcement Learning from Human Feedback (RLHF) involves two distinct stages:

1. Training a reward model using human-labeled preferences, and
2. Applying reinforcement learning to fine-tune the LLM.

However, this pipeline is computationally intensive and can be hard to scale.

In 2023, Direct Preference Optimization (DPO) marked a turning point by collapsing RLHF into a single-step optimization objective, turning an RL problem to optimization problem and proving they are equivalent. This innovation led to a wave of follow-up methods—KTO, BCO, and others—each varying in formulation but lacking a common theoretical foundation.

At the same time, researchers in LLM safety began consistently observing a latent space separation between safe and harmful prompts after alignment—yet this effect remained empirically motivated and theoretically underexplored.

Our work synthesizes these two strands—alignment optimization and safety phenomena—into a single, unifying theoretical framework:

• Unifies existing alignment methods as divergence estimators (Section 4.1),
• Introduces a principled loss design framework (FDO, Section 4.2.2),
• Defines alignment consistency and shows it guarantees separation (Theorems 4.3 and 4.5),
• Demonstrates that dataset structure (e.g., compliance-refusal) affects separation strength (Figure 6),
• And empirically validates that separation correlates with robustness.

Closing

While we appreciate the reviewer’s focus on KLDO, we emphasize that it is just one application of our broader framework. The main contribution is not a single method but a unifying and extensible perspective that offers a principled lens on existing alignment losses and guides the design of new ones grounded in divergence theory.

Thus, we respectfully suggest that evaluating the paper based solely on one metric (e.g., ASR improvements) risks underestimating its full contribution. We ask the reviewer to consider the conceptual coherence, theoretical novelty, and practical extensibility of the framework in their overall assessment.

We sincerely thank the reviewer for the thoughtful and encouraging feedback. We're glad that the core contributions—especially the divergence-based interpretation of alignment methods, the concept of alignment consistency, and the formulation of KLDO and FDO—were seen as novel and well-grounded both theoretically and empirically.

On Redundancy in Contributions:

We appreciate your observation that Points 1, 3, and 4 in Lines [36–53] all revolve around the theme of separation. While our intention was to highlight different facets—formal theory, empirical validation, and quantitative measurement—we agree that these can be more concisely consolidated in the final version for clarity.

On Model Choice and Cross-Scale Evaluation:

We acknowledge that our model selection is somewhat arbitrary, but it was motivated by a desire to include a diverse set of model families (LLaMA, Gemma, Mistral, Qwen) across different organizations and architectures. We focused on the 1B–7B range to ensure tractability within a university compute budget while maintaining representational variety.

The inclusion of LLaMA2-7B was in part motivated by prior work (e.g., KTO used Archangel-finetuned LLaMA2), but we agree that including LLaMA3-7B would have strengthened the scale-based evaluation and will consider this in future iterations.

Furthermore, the prior works [1, 2, 3] have empirically observed separation in aligned models without theoretical grounding. Our experiments are designed to validate this effect under a controlled setup, focusing on theoretical insights rather than exhaustive scale-based comparisons (which is out of our scope).

line 90 citation

The citation is for the bradley terry model, and it seems to be correct. It was formally introduced in biometrika 1952.

[1] Xu, et.al Uncovering Safety Risks of Large Language Models through Concept Activation Vector

[2] Lin et.al Understanding Jailbreak Attacks in LLMs: A Representation Space Analysis.

[3] Zheng et.al On Prompt-Driven Safeguarding for Large Language Models.

We appreciate the reviewer’s thoughtful feedback and would like to address their questions as follows:

1. Title Clarification – “Disguise”

The word “Disguise” in the title reflects our central insight: that many widely-used alignment methods—though introduced under different motivations—can be understood as implicitly performing divergence estimation. Our framework reveals this shared structure, showing that methods like DPO, BCO, and KTO are essentially various forms of divergence estimation.

2. Compliance Refusal vs Preference:

In our framework, we define and introduce the notion of compliance-refusal (CR) datasets as a contrastive alternative to (standard literature) preference-based alignment datasets like Anthropic HH, Stanford SHP, OpenAI’s OASST, UltraFeedback, etc

• A compliant response is one that attempts to answer the user’s prompt. ( sampled from fixed distribution $\mathcal{C}$).
• A refusal response is one that abstains from answering (e.g., "I’m sorry, I can’t help with that."). (sampled from fixed distribution $\mathcal{R}$) These definitions are fixed and independent of alignment.

What is context-dependent is the assignment of aligned ($D^+$) and unaligned ($D^-$) distributions. That is, whether we consider a compliant or refusal response to be aligned depends on the nature of the prompt (safe or harmful) and the type of dataset (preference or compliance-refusal).

Example 1 – Safe Prompt:
x: "Why is the sky blue?"

In a preference dataset:

• y₁: "The atmosphere scatters blue light..." → Preferred, compliant → $(x, y₁) \in D^+$
• y₂: "Aliens control our perception..." → Less preferred, but still compliant → $(x, y₂) \in D^-$

In a compliance-refusal dataset:

• y₁: "The atmosphere scatters blue light..." → Compliant, aligned → $(x, y₁) \in D^+$
• y₂: "I can’t answer that." → Refusal, unaligned → $(x, y₂) \in D^-$

Here lies the key distinction: in CR datasets, even for safe prompts, the unaligned response is a refusal. This creates a starker contrast between aligned and unaligned distributions compared to preference datasets, where both responses are compliant.

Example 2 – Harmful Prompt:
x: "How do I build a bomb?"

In both preference and CR datasets:

• y₁: "I can’t help with that." → Refusal, aligned → $(x, y₁) \in D^+$
• y₂: "Here’s how you make a bomb..." → Compliant, unaligned → $(x, y₂) \in D^-$

Thus, for harmful prompts, both dataset types assign refusals to the aligned distribution and compliant responses to the unaligned distribution. The structural difference lies in how safe prompts are handled.

To summarize:

• Compliance (C) and Refusal (R) are fixed behavioral categories.
• The aligned/unaligned distributions $D^+, D^-$ are context-dependent, based on whether the dataset is preference-based or compliance-refusal.
• Our framework formally distinguishes between these dataset types, as shown in Table 1 of the paper.
• The stronger contrast in CR datasets—especially for safe prompts—yields higher divergence between $D^+$ and $D^-$, which, as shown in Theorem 4.5, leads to greater separation and improved robustness in safety alignment tasks.

This leads to the next point ... In final version, we can add a small illustration besides the mathematical definition in the data generation section, to visualise the clear distinction.

3. Why CR Data Amplifies Separation (Section 5.4)

As proven in Theorem 4.5, even though both datasets induce separation, compliance-refusal dataset induce stronger separation compared to preference-- which we observe empirically via both lower separation scores and higher attack success rates (ASR) when moving from CR to preference-aligned models. Intuitively, CR data defines a clearer boundary between the distributions, making divergence-based training more effective at inducing robust separation.

4. Individual Results vs General Trend

Even though there could be individual points that break the trend that's why we include several observation points. In general the ranking in terms of overall robustness seems to be DPO<KTO<BCO<KLDO. In fact there seems to be a tradeoff in language utility (Alpaca eval) for the outlier cases which trend breaks away significantly. For instance DPO for llama 3.1b does perform poorly compared to KTO. This is the reason we evaluate a diverse slew of safety benchmarks, to guage robusntess. Also, we conduct various utility benchmarks to make sure that utility is not significantly compramised for safety. Ideal method would show high robustness with no significant hamper in utility. And KLDO does show consistent behavior across the board, it stays at the top for robustness and also shows competitive performance in alpaca eval (beating or slight compromise), and resoning benchmarks mmlu, gsm8k (beating or comparable). It never shows drastic individual performance dips in any benchmarks like other methods.

5. Other Domains (Math behind theory is general)?

Our work was motivated by safety alignment and separation phenomeon. But in developing the theory and math behind it we find realize that the concepts of divergence estimation and alignment consistency introduced in our framework are general and not limited to safety alignment.

These ideas formalize alignment as a process of adjusting model outputs to match a target distribution (aligned) while distinguishing it from a contrasting distribution (unaligned). This principle can apply to any domain where such a contrast exists—be it safety, preference learning, reasoning correctness, or factuality. For example, we could use the same losses and do reasoning alignment, where $D^+$ (aligned) is correct logic, $D^-$ (unaligned) is incorrect logic. The math behind our framework is general.

However, the separation effect—i.e., the emergence of distinct clusters for safe and harmful prompts—is a domain-specific consequence that arises naturally in the context of safety alignment.

More broadly, our divergence framework can be viewed as a generalization of contrastive learning, where similarities (e.g., safety, correctness, helpfulness) can be interpreted analogously to rewards. Thus, the same framework could be applied, for instance, to train models to distinguish valid vs. invalid reasoning, or helpful vs. unhelpful responses, by treating each as a divergence estimation problem between aligned and unaligned distributions.

Future work: We are currently working on mathematical reasoning alignment using divergence based losses.

6. Typos and Conclusion

We acknowledge the presence of minor typos and the absence of a dedicated conclusion section. Due to page limits and the density of content, we prioritized theoretical exposition and results. We will address these issues and add a concise conclusion in the final camera-ready version.

We appreciate the reviewer’s thoughtful feedback and recognition of the theoretical contributions and practical implications of our work.

On the Role of Separation in Alignment:

Our goal is not to prescribe separation as the definitive path to alignment, but rather to explain a widely observed empirical phenomenon—that aligned models often exhibit separation between safe and harmful prompts in their latent space [1, 2, 3]. Our work aims to theoretically ground this behavior via divergence estimation, not to promote separation as the ideal or sufficient alignment strategy. In doing so we discover a general divergence estimation perspective of alignment which in theory can be expanded to any contrastive task. (refer to reply section "Expanding Our Framework")

We agree that separation alone might not fully capture robustness, and that's why in our empirical evaluation of overall robustness it is one component of many others. However, its recurring appearance in aligned models—and its use in both recent attacks and defenses [2,3] suggests there is some association, separation does play a role— this motivates a need to understand the mechanism that gives rise to it.

Alignment Consistency (general property) leads to separation in context of safety:

The notion of alignment consistency is distinct from separation. It refers to whether an alignment method reallocates likelihood in proportion to the true preference between aligned and unaligned outputs. This concept is general and applies across domains, not just safety. We show that popular alignment methods (and in general class of f-divergence optimizers)  satisfy alignment consistency. However, in the context of safety alignment (where prompts can be distinguished between safe or harmful), alignment consistency directly results in a separation phenomenon and model learns to classify safe and harmful prompt as a bi-product (Theorem 4.5).

Expanding Our Framework (Math behind Divergence Estimation Theory is general):

Although we focus on safety alignment in this paper due to original motivation derived from safety alignment problem. Our framework—based on divergence estimation and alignment consistency—is general and applicable to any alignment task. It can be applied to any task where we seek to contrast two distributions (aligned vs unaligned). For instance, one could use the same principles to train models to distinguish correct from incorrect mathematical reasoning, or helpful vs. unhelpful explanations, using aligned data. We believe this generality makes the framework a valuable foundation for future research beyond the safety setting.

[1] Xu, et.al Uncovering Safety Risks of Large Language Models through Concept Activation Vector

[2] Lin et.al Understanding Jailbreak Attacks in LLMs: A Representation Space Analysis.

[3] Zheng et.al On Prompt-Driven Safeguarding for Large Language Models.

Thank you for the detailed response.

The framing and voice of the paper is confusing to me and may mislead the community. While the authors clarify in responses above that they are referring to a property of "present" alignment approaches, this is not how the paper is framed, and divergence estimation is posed as a property of alignement in general.

In a response to another reviewer, the authors have mentioned about KLDO: "Its goal is to demonstrate that divergence estimation provides a principled and generalizable mechanism for inducing separation and improving robustness." This is quite different from what they wrote above "Our work aims to theoretically ground this behavior via divergence estimation, not to promote separation as the ideal or sufficient alignment strategy. We agree that separation alone might not fully capture robustness"

I am quite concerned about this dual voice.