Thank you for your feedback.

The translation is based on a set of fixed rules, making it unclear how general these rules are.

See comment below about WMC.

The algorithm uses rules in table 6, which covers P1*P2, (1-P1) and P1+P2. I wondered how general this is. For example, how can sqrt(P1×P2) be translated?

Given that our analysis is based on WMC, the losses being decompiled must be expressible as the polynomial class defined by WMC; by construction, we restrict this to the disjoint multinear polynomials defined on line 348. Except for DPOP (see discussion below), this naturally captures all known preference losses, including those in Table 2, which we emphasize is a fairly comprehensive set of DPA losses (we updated our appendix to include additional DPO variants that fit this analysis), as well as the common loss functions such as cross-entropy and unlikelihood.

The square root clearly does not fit this equation class, so it is not covered by our analysis. Is there a particular loss involving a square root that you have or mind, or some other salient loss that seems out of scope?

The method is limited to the "main" component of the loss, disregarding the regularisation terms. Regularisation terms usually play important roles in those preference optimisation losses. So while I believe this method is able to interpret part of the semantics behind the loss, it may also ignore important aspects.

This is an important point, which we will say more about in the updated draft. We note, however, that the decision to avoid regularization terms in our analysis follows other formal studies such as by Tang et al. (2024). We also found cross-entropy terms, even when removed, to have limited impact on empirical performance, which is a finding consistent with [1].

[1] Hanyang Zhao et al. RainboPO: A Unified Framework for Combining Improvements in Preference Optimization

Special treatments are required for some DPA loss variants (e.g., DPOP).

This special treatment is due to the compositional constraint from Assumption 1 (inspired from programming semantics), which stipulates that every model prediction in a loss should be directly accounted for in the semantics. For clarity: the DPOP loss has terms such as $p_{\theta}(y_{w} \mid x)^{k}$ and $p_{ref}(y_{w} \mid x)^{k}$ with exponents $k$ that make the equation not multilinear when $k > 1$ (without loss of generality we only considered cases where $k=1,2$). Our decision to make them multilinear when $k >1$, by creating new variables $p_{\theta2}(y_{w} \mid x)$ and $p_{ref2}(y_{w} \mid x)$ is based on a common polynomial transformation for making polynomials multilinear, which in the end maintains compositionality. Importantly, as we also detail in Appendix F, the value of $k$ also varies from the instance to instance, so both $p_{\theta2}(y_{w} \mid x)$ and $p_{ref2}(y_{w} \mid x)$ do not have a fix value (i.e., they either can take the value of the other variables or be equal to 1 when $k=1$), which we think justifies treating them as semantically distinct values.

We will discuss this more in the draft, including making this more clear when we first reference DPOP around lines 140 (second column). We acknowledge that such exponents could be treated differently, e.g., as parameters that are part of the probability computation for each variable similar to how length normalization is computed, which would sidestep completely these issues involving multilinearity.

Furthermore, it is unclear to me how the proposed formulation can help explain the performance of different DPA variants, although from figure 4 there are indeed relations and differences in the semantics of these variants.

As we discussed in our response to JCXg , Proposition 3 establishes that semantic entailment induces certain monotonicity properties w.r.t to the behavior of the compiled losses, and provides us with a notion of the relative constrainedness of losses that is grounded in loss behavior. As discussed in Sec.E.1, the relative constrainedness of a loss is a property that we think has an important impact on its empirical performance, which is explained by our framework and observable when looking at training dynamics as shown in Figure 8.

We moved this analysis into the appendix due to space issues. If accepted, we intend to include some of this empirical analysis in the main paper (including some additional experiments that scale the reported ones).

Thank you for your encouraging feedback.

How well can other researchers leverage the proposed framework to identify novel loss formulations beyond the 4 shown in the paper?

While we reported experiments on 4 losses, we derived many more novel losses, including the 16 single model losses shown in Figure 7 each (excluding 1) with a novel DPO variant. We think that this set exhaustively captures the full set of single model losses that researchers would want to experiment with (it exhaustively shows all definable losses between CEUnl and unCPO), and offers many novel DPO variants that have not yet received empirical verification. We plan to release all the associated code and loss implementations to facilitate further work in this area.

Given that our framework now allows one to define losses using a much more expressive logical language, where any valid propositional formula constitutes a valid loss function, we also believe that this makes it possible to more easily devise entirely new classes of loss functions that would otherwise be difficult to derive working solely from the mathematics of DPO (e.g., losses that involve more complex forms of feedback, non-differentiable components).

We will include a concrete example of this in the updated draft.

Does this approach generalize to other models and larger parameter counts

These approaches can be generalized to any language model of any size or number of parameters; the size of the underlying language model is usually not a critical factor in the loss computation (see below).

Is there any significant computational overhead with the new loss formulations? (especially cCPO)

No. The cCPO loss is computed according to the equation given in Appendix D, and does not require any more computation than another of the other losses in that table. As above, computing losses is a relatively easy computation once the basic forward calls have been made to the model to obtain model output probabilities, which are done independently of the loss computation.

Related to the point above, Appendix E contains many important results to validate the proposed frameworks’ capability.

Thank you for this suggestion. If accepted we intend to use the additional page to move into the main paper some of the details of the experiments now in the appendix (these were not included due to space limitations), as well as subsequent experiments we did to further verify these findings (see our response to JCXg).

Thank you, we are excited that you find our work “really exciting”, below we address your comments and concerns.

However, while the paper briefly mentions new loss variants, it lacks a strong empirical demonstration

Please see our response below. Much of our experimental results were pushed to the appendix due to space limitations. If accepted we intend to use the extra page to include more empirical results in the main paper, including the extra experiments we mention below.

computational complexity of WMC

Indeed, WMC is a hard algorithmic problem, but we note that in our study this complexity is side-stepped since we are working with problems with a small number of variables (2-4). In such cases, one can simply write the WMC formulas in full and simplify them using algebra offline, which is how we obtained the new losses in App.D (we updated the draft to include details of how to compute this using SymPy).

Nonetheless, the probabilistic reasoning community has devised various “knowledge compilation” techniques that allow one to scale WMC to much larger problems by compiling these logical representations into tractable circuits, which make many problems feasible in practice (and sometimes in theory). We are confident that if we were to greatly expand the complexity of our programs, such techniques, or other advanced SAT techniques, could be leveraged for doing the involved experiments.

Have you discovered any novel loss functions that outperform existing approaches, and more importantly, a mechanism

Yes, we found in the experiments reported in Table 5 that the novel loss, $\ell_{cCPO}$ from Figure 4 outperforms the known loss $\ell_{CPO}$ in a win-rate study adapted from Meng et al 2024. This does show evidence that our theory is able to yield competitive new losses (we have subsequently run these same experiments at larger scales using a >3x larger Smollm-1.7B model and are seeing similar trends; we intend to report these results in the updated draft pending the full results).

In terms of mechanisms, we think our experiments do give insight into this, as noted below.

– We find that the logical constrainedness of a loss function is an important contributing factor to its empirical success. For example, the highly unconstrained losses in Fig.4 tend to have spurious behavior due to the nature of their semantics and how the underlying constraints can be satisfied (see discussion in E.1), which we can see clearly in the behavior of their win/lose log probabilities and training dynamics shown in Figure 8 (such empirical behavior is something we consistently see across the many experiments we’ve run at different scales).

Since DPO was introduced, there has been much puzzlement about the empirical behavior of log probabilities during training, which we think our constraint satisfaction view of the problem can help to elucidate.

– We also see in Table 5 that losses perform differently on different subsets of data, which is a trend that we’ve seen persist through our later experiments. We take this as evidence that different tasks and datasets involve different semantics, requiring one to carefully tailor their losses to those semantics.

How do we then navigate such a huge space

We tried to address this in “How is the loss space structured” (the right column of line 287). While the space is indeed large, we can see through results such as Proposition 3 that loss behavior is linked in interesting ways to the logical semantics of the losses, which can be exploited for exploration. The strategy we pursued in our case study was the following: start with an empirically successful loss (e.g., CPO), formalize its semantics, then modify its systems to find new losses that are either more constrained (or that entail CPO) or less constrained (that are entailed by CPO) and experiment accordingly.

We think strategies like this are useful tools for navigating this space.

My main concern is regarding the notion of using an absolute value

The \epsilon notion is only meant to be a tool or heuristic to conceptually think of the distribution as digitized or divided into “valid” and “invalid” outputs; it is not a value that we model explicitly or that plays any role in our formal analysis, and hence we do not make any assumptions about it (e.g., it needn’t be a fixed value).

Regarding your example: if we interpret the top symbolic formula in Figure 2 in terms of this digitized distribution, this just says that “whenever we find the loser to be a valid generation (i.e., to be above this \epsilon) we should always find the winner to also be above this line (\epsilon) too”. Note that this semantics, which underlies many DPA approaches, does not rule out the possibility that the loser is a “valid” generation (or also above \epsilon). So for your example of (x, y_1, y_2) (x, y_2, y_3) we could set this hypothetical \epsilon to be the probability y_3 while still satisfying this constraint.

Thank you for your feedback, we are pleased that you find our approach to be “a valuable addition to the relevant science literature”

many relevant papers which incorporate "scalar reward signals" in the training objective. With the fine-grained reward information, those methods tend to outperform methods relying on binary preference labels.

Thank you for these pointers. This does look like an exciting new area where we can try to apply our techniques. While we haven’t digested yet all the details of these papers, it does seem like the kind of distillation objective in Fisch et al. 2025 (Eq. 7) could fit into our framework by adding these additional reward model estimates into our logical formulas as additional predicates (we will think more about this and, if appropriate, mention this direction in an updated draft).

Thank you for the feedback; as noted in our response to the above reviewer, we’ve already taken steps to improve the presentation, which we hope will also address your concerns.

We are encouraged that you nonetheless found our approach to be “compelling” and an “interesting avenue to explaining, manipulating, or exploring losses”

Below we address your particular comments and questions.

the experiment mostly shows a basic usage example without providing deeper insights into the method or discussion on limitations.

Our main contribution is to provide clarity into the ever-growing space of preference losses. The symbolic view of these losses allows us to make the preferences encoded by them explicit, understand relationships between them, and, via counting arguments, enumerate the space of all possible losses with or without a reference model. Moreover, it also provides a conceptual framework for inventing novel kinds of losses.

We also note that much of our experimental results were pushed to the Appendix due space issues (see please our response to JCXg). If accepted, we intend to use the additional page to incorporate such results in the main paper.

"In particular, we focus on approaches based on probabilistic logic.". The next sentence disagrees, "In contrast, we focus on the inverse [...]".

Sorry for the confusion, we will modify this for clarity.

To clarify: the point about this “inverse problem” is that while most people in the neuro-symbolic field have focused on the problem of “compilation” (i.e., translating symbolic formulas into loss functions by interpreting those formulas using probabilistic logic), we focus on the “inverse” and more unique problem of “decompilation” (i.e., deriving symbolic formulas for existing loss functions that we also interpret in terms of probabilistic logic).

In both cases, probabilistic logic is used as the ingredient that makes communicating between these symbolic forms and losses possible, which is why we say that our approach is “based on probabilistic logic” (we felt that this was important to clarify since other popular modes of translation, such as fuzzy logic, could have been used here as an alternative to the probabilistic approach).

On page 3, line 146, "No reference" is in bold text, this might be by mistake.

We will fix this inconsistency.

Figure 1 illustrates the core idea of the paper well, but the concepts within the symbolic program may be difficult to parse for a first-time reader on page 1

As noted in the response to fMQw, we completely revamped this figure to make it more clear along the lines that you suggest.

The abstract employs the abbreviation DPO without introducing it beforehand..

Thank you for catching this, we will change DPO to “Direct Preference Optimization”.

Figure 1 illustrates the core idea of the paper well, but the concepts within the symbolic program may be difficult to parse for a first-time reader on page 1. Perhaps additional annotation or a simplified illustration/example would be more digestible at this point in the paper.

Please see our response above.

The paper's presentation suffers from how information is distributed throughout the manuscript. For example, on page 5, "Decompilation into semantic loss" is described but references/requires insights from Table 2 (page 3), Section 5.2 (page 7), and Table 6 (page 11) to be understood.

We updated our draft to account for these structural issues.

Thank you for your feedback, we are encouraged that you found our approach to be “very original and promising”. We have already taken steps to improve the presentation in the places you mention and we will address your particular points below.

To me, the start of Sec. 3 is harder to read than necessary. It might help to introduce the role of β a bit earlier and move the information from the caption of Tab. 2 ("All losses ...") to the main text to make it self-contained.

We will address this in the revised draft.

Fig. 1: DOP2 could also be named as such on the right-hand side.

We have already modified Figure 1 according to your suggestions and in addition: 1) tried to make the figure easier to read and less blurry and; 2) we expanded the caption to more clearly state the problem and the goals of the paper.

Fig. 1 talks about "compilation" and "derivation"

Thank you for noticing this. We will fix it to be more consistent.

Also, we restructured part of Section 4 to make clearer the precise relationship between compilation and derivation (i.e., that we treat the former as the inverse of the latter), which we hope better motivates why we immediately start in Sec. 4.1 by talking about the “Compilation to semantic loss”.

Fig. 3: The entire figure is a bit unclear, and it personally confused me more than it helped. I

As with Figure 1, we did a complete revamp of this Figure, which we hope addresses your confusion and makes the modifications you suggest.

To state the role of Figure 3 more clearly (as we do in our updated draft): Preference structures, via Prop. 2, can be equivalently expressed in terms of two Boolean functions, $P_w$ and $P_l$, which correspond to the checkmarks and xmarks in the figure respectively (and are also equivalent to the formula forms in Eq. 4). We think that this view is helpful for: 1) understanding the corresponding model counting problem visually; 2) understanding how our generalized formulation of semantic loss, as captured by the equation in this figure, is general enough to capture things like conditional probabilities (i.e., not satisfying $P_C$, as captured by the white boxes in Fig 3; we will update the arrow from $P_C$ in Fig 3 to point to such white boxes without any checkmark or xmark).

"We assume that all preference loss functions have an internal logic that can be expressed in the form described above." To what degree is this a limitation of the approach? What happens if that is not the case and we apply the translation regardless?

To clarify this claim, which is quite general, we assume that all preference losses have some internal logic that can be expressed in a discrete form. Whether or not they can all be expressed in the logic we propose is a separate question (e.g., it might be that some preference losses outside of our study require a more complex logical system beyond propositional logic. We will clarify this point, but we think it is an important working assumption to note).

For the preference losses under consideration in Table 2 (which cover many of the most popular DPA losses), Thrm. 2, however, does establish that our decompilation procedure and logic correctly captures the logic of these losses in the following sense (which we will make more clear): our decompilation procedure can take any of these losses as input and produce a semantic representation that can be compiled back into exactly and uniquely that loss via our logic.

The main condition that needs to be met for the decompilation procedure to be applicable is that the input loss equation is a “disjoint multilinear polynomial” as defined in line 346 (regarding your other point, we see that a non-inline version of this would be helpful here to make this point more clear). Of course, if the input loss does not follow this polynomial form the translation runs the risk of not being correct, but such losses are out of scope for this study and we could imagine making the translation more complex to expand this to other polynomial classes.

How is, intuitively, the implication of the left-hand side of, say Fig. 1, realized by the loss formula?

Yes, we agree that a clear intuition here would be helpful. Given the way that our implication construction works (i.e., the construction in the proof of Prop.2 which drives Algorithm 1 and our decompilation procedure), the left side of the implication in semantic representations like those in Figure 1 (or the representations in Table 4) corresponds to the lower/bottom part of the log ratios in the preference losses (Table 2, $\rho_{\theta}^{b}$) and the right side to the upper part of these log ratios ($\rho_{\theta}^{t}$).

The definitions are mostly given inline in the text

We will fix this, particularly by considering making some of the core definitions that our formal results rely on (non-inlined) formalized definitions.