Aligning Compound AI Systems via System-level DPO

We sincerely thank the reviewer for the constructive and encouraging feedback. We will incorporate clarifications and revisions, as well as new experiments, in the camera-ready version. Details regarding your comments follow.

• Novelty and related works.

  • We thank the reviewer for the positive assessment and the request to clarify novelty and related work. Our contribution is the first framework that jointly aligns the parameters of an entire compound AI system, modelled as a Directed Acyclic Graph (DAG). This formulation lets us derive a DPO-style loss, and we prove the first alignment guarantee for compound AI systems. Different from our scope of parameter-level alignment, prior work primarily explores prompt-based coordination, which is an orthogonal line of research. It is an exciting future work to investigate how the two orthogonal directions interplay with each other, and how to effectively combine them.
  • The resulting algorithm being conceptually simple is a result of its generality: it can be applied to any compound AI systems with a DAG factorization. Although the general scheme is simple, our implementations involve non-trivial procedures such as a tailored derivation of the tractable loss function for the LLM+Diffusion model system (Appendix D), and diverse sampling strategies used in both SysDPO-direct (Appendix F) and SysDPO-Sampling (Appendix B). Without such diversity-encouraged sampling procedures, MC sampling yields sub-optimal performance, as shown next.

• How the intermediate samples are generated and how their quality affects the final performance. We would like to thank the reviewer for suggesting a more detailed explanation, as this is indeed a key factor for SysDPO-Sampling. We use Diverse Beam Search (DBS) for SysDPO Sampling (detailed in Appendix B), with a goal to improve diversity among the intermediate samples. We conduct a new ablation study to illustrate that the quality and diversity of those samples matter a lot. We begin by comparing Diverse Beam Search (DBS) with Monte‑Carlo (MC) sampling and then examine how performance scales with the number of intermediate candidates sampled during training. Our experiments reveal two clear trends: (i) within the tested range, increasing the sampling budget $k$ has little to no impact on overall performance; and (ii) DBS achieves higher win‑rates under the same sample count.

  • Experimental Settings and sample details. Building on the LLM collaboration system in section 5.2, each training instance starts from input $x$, model $\psi_1$, and $\psi_2$. During training, we sample $k$ intermediate candidates $y^1, \dots y^k \sim p_{\psi_1}(\cdot | x)$ and leverage them to calculate the loss that jointly optimize $\psi_1, \psi_2$. At evaluation time, we eliminate sampling stochasticity by setting the temperature to 0 and deterministically selecting $y^\* = \max_y p_{\psi_1}(y | x)$ and $z^\* = \max_z p_{\psi_2}(z | x)$. We evaluate the system performance based on the quality of $z^\*$.

  • Sampling methods. For DBS, We obtain $k$ candidates $y^1, \dots, y^k$ using $k$ beam groups and a diversity penalty of 20; we report results for $k=2,4$. For MC sampling, we independently sample $y^1, \dots, y^k$ with temperature 1.0 under different random seeds, evaluating $k=2,3,4,5.$

  • Results. The table below summarizes the win‑rate of each configuration. DBS outperforms MC with fewer number of sampled candidates. Both the methods are consistent with varying $k$ within the test range, suggesting that 2 samples may be sufficient without drastically increasing the budget. Compared to DBS, MC often draws candidates that are near‑duplicates.

    Method	k	WR-Prompted (%)
    DBS	2	68.5
    DBS	4	68.2
    MC	2	66.8
    MC	3	67.0
    MC	4	66.7
    MC	5	66.0

  • Examples of the intermediate samples. Here are some examples of intermediate outputs generated using different sampling methods for the same questions.

    • Case 1:

    Question: “Extract the answer to the following question from the movie plot. If the question isn't answerable, please output "Can't answer ……”:

    DBS Sampling:

    $y^1$: Haymitch Abernathy's arm.

    $y^2$: Can't answer. The question is not answerable as it does not provide a specific scene or moment in the movie where Johanna cuts from Katniss's arm. The plot …

    MC Sampling:

    $y^1$: Haymitch Abernathy.

    $y^2$: Haymitch Abernathy.

    • Case 2:

    Question: Here is some data about a restaurant: name = The Golden Palace, eatType = coffee shop, food = French, priceRange = £20-25, customer rating = high, area = city centre. Write a sentence that includes the following data about a restaurant

    DBS Sampling:

    $y^1$: The Golden Palace is a fancy coffee shop in the city centre that serves delicious French food for a price range of £20-25, and it has a high customer rating, making it a great place to enjoy a meal with friends or family.

    $y^2$: There is a restaurant called The Golden Palace located right in the city centre, where they serve delicious French cuisine at a price range of £18-22 per person, and customers have given it a high rating for its excellent service and cozy atmosphere.

    MC Sampling:

    $y^1$: The Golden Palace is a fancy coffee shop located in the heart of the city center, serving delicious French cuisine for around £20-25 per person, with a high rating from satisfied customers who love dining there.

    $y^2$: The Golden Palace is a fancy coffee shop located in the heart of the city centre, where they serve delicious French cuisine at a moderate price range of £20-25 per person, and customers highly rate their experience.

    The MC samples are nearly identical, whereas DBS supplies two semantically distinct options. This illustrates how DBS enhances sampling diversity and, in turn, system performance.

• A baseline of DPO with Monte-Carlo sampling. We would like to thank the reviewer for suggesting this experiment. This new ablation study, as shown above, shows that MC sampling underperforms DBS even with more samples. We start the sample size with $k=2$, as a single sample $k=1$ would have the probability terms $p_{\theta_i}(y^\alpha_i\mid `Pa`(y^\alpha_i))$ canceled out in the SysDPO-Sampling (Eq. 9 in Appendix B).

We would like to once again thank the reviewer for their constructive comments and for acknowledging the importance of aligning compound AI systems. We hope these clarifications and additions will strengthen the paper and more clearly convey its contributions to the study of alignment in compound AI systems.

We sincerely thank the reviewer for the constructive and encouraging feedback. We will incorporate clarifications and revisions, as well as new experiments, in the camera-ready version. Details follow.

• Scalability, efficiency, and computational cost.

  • We thank the reviewer for highlighting efficiency and scalability. This is indeed a challenge in our method and also a fundamental challenge of compound AI system alignment. The fundamental challenge is due to how the preference signal is collected: the user preference is only given to the final output $z$ but not any intermediate outputs $y$. In other words, it would be much easier if we had a preference oracle to compare two intermediate outputs directly, i.e., $`pref`(y^1\succ y^2\mid x)$, from which running standard DPO separately on each model would suffice. Unfortunately, this is ill-defined because the evaluation of a component in a compound system must depend on other components, i.e., there is no ground-truth preference oracle of the intermediate outputs that stands independent of the generative system itself. Therefore, as you rightly point out, intermediate samples have to be drawn online constantly (SysDPO-Sampling) or have sufficient coverage (SysDPO-Direct). The challenge is thus fundamentally a credit assignment problem under non‑differentiable, hidden intermediate interactions, and the fact that humans only have access to and give preferences to the final outputs.

  • How we tackle the efficiency challenge. The core idea of improving the sample efficiency of the intermediate outputs is diversity. For SysDPO-Direct, we adopt a set of diverse styles of prompting to construct the preference dataset (detailed in Appendix F). For SysDPO-Sampling, we adopt diverse beam search (DBS) to improve the diversity of the intermediate outputs.

    • We run a new ablation study to show that DBS outperforms Monte Carlo sampling with the same sample budget, highlighting the importance of diversity for efficiency. We begin by comparing Diverse Beam Search (DBS) with Monte‑Carlo (MC) sampling and then examine how performance scales with the number of intermediate candidates sampled during training. Our experiments reveal two clear trends: (i) within the tested range, increasing the sampling budget $k$ has little to no impact on overall performance; and (ii) DBS achieves higher win‑rates than MC with the same sample count.
    • Experimental Settings and sample details. Building on the LLM collaboration system in section 5.2, each training instance starts from input $x$, model $\psi_1$, and $\psi_2$. During training, we sample $k$ intermediate candidates $y^1, \dots y^k \sim p_{\psi_1}(\cdot | x)$ and leverage them to calculate the loss that jointly optimize $\psi_1, \psi_2$. At evaluation time, we eliminate sampling stochasticity by setting the temperature to 0 and deterministically selecting $y^\* = \max_y p_{\psi_1}(y | x)$ and $z^\* = \max_z p_{\psi_2}(z | x)$. We evaluate the system performance based on the quality of $z^\*$.
    • Sampling methods. For DBS, We obtain $k$ candidates $y^1, \dots, y^k$ using $k$ beam groups and a diversity penalty of 20; we report results for $k=2,4$. For MC sampling, we independently sample $y^1, \dots, y^k$ with temperature 1.0 under different random seeds, evaluating $k=\{2,3,4,5\}$.
    • Results. The table below summarizes the win‑rate of each configuration. DBS outperforms MC with the same number of sampled candidates. Both the methods are consistent with varying $k$ within the test range, suggesting that 2 samples may be sufficient without drastically increasing the budget. Compared to DBS, MC often draws candidates that are near‑duplicates.

    Method	k	WR-Prompted (%)
    DBS	2	68.5
    DBS	4	68.2
    MC	2	66.8
    MC	3	67.0
    MC	4	66.7
    MC	5	66.0

  • Computational cost. Aligning the system of two LLMs jointly took approximately 30 hours on a single NVIDIA H200 GPU. We acknowledge that broader scaling is out of our limited academic budget and remains open. We therefore see this work as a principled first step that lays the groundwork for future large-scale investigations.

• Fixed DAG structure.

  • We appreciate the reviewer’s insight and agree that many real-world systems feature dynamic, input-dependent control flow. For settings with loops or dynamic branches, we envision two extensions. (i) time-unrolling for recurrent interactions. For systems with loops, one can unroll the interaction over time, yielding a finite-horizon DAG, similar to back-propagation through time. (ii) stochastic masking. Some nodes may learn a policy over which downstream modules to invoke, where the random variable is a mask over the node’s out edges. Its probability naturally integrates into the SysDPO framework. Exploring these directions is an exciting future work.

• The role of $\beta$ in $\beta$-perfect alignment.

  • It is very interesting how $\beta$-perfect alignment can be precisely interpreted from many different perspectives. However, we acknowledge that the explanation of the original manuscript can be improved for better clarity. Below, we first give the intuition of $\beta$-perfect alignment. Then, we discuss how the $\beta$ is the same as the $\beta$ appearing in the DPO/RLHF objective as the KL divergence.
    • $\beta$-perfect alignment of generative models. Given an input $x$, consider two versions of the generated output $z^1$and $z^2$. Suppose there is a ground truth preference oracle $`pref`(z^1\succ z^2\mid x)\in (0, 1)$, which states how much $z^1$ is preferred over $z^2$. The model $\theta^\*$ is $\beta$-perfectly aligned to the preference oracle $`pref`(\cdot)$ if for any $x\in\mathcal{X}$ and $z^1, z^2 \in \mathcal{Z}$ such that $`pref`(z^1\succ z^2\mid x) = \frac{p_{\theta^\*}(z^1\mid x)^\beta}{p_{\theta^\*}(z^1\mid x)^\beta+p_{\theta^\*}(z^2\mid x)^\beta} \qquad\qquad{(\text{Definition 1})}$
    • What varying $\beta$ does. We can understand the effect of varying $\beta$ through a concrete example. Suppose we have a preference oracle of $`pref`(z^1\succ z^2\mid x)=0.8$ on data $(x, z^1, z^2)$. Then, we obtain a $\beta$-perfectly aligned model $\theta^\*$ that satisfies the alignment equation (Definition 1). We may rearrange the alignment equation (Definition 1) to be
    $$$$

  pref(z^1\succ z^2\mid x) = \frac{1}{1+\left(\tfrac{p_{\theta^*}(z^2\mid x)}{p_{\theta^*}(z^1\mid x)}\right)^{\beta}} $$ If $\beta\ll 1$ is close to $0$, it would require the ratio $p_{\theta^\*}(z^2\mid x)/p_{\theta^\*}(z^1\mid x)\ll 1$ for the equation to hold: the aligned model almost always output the most preferred result deterministically. Conversely, if $\beta \gg 1$, it would require the ration $p_{\theta^\*}(z^2\mid x)/p_{\theta^\*}(z^1\mid x)\approx 1$, i.e., approaching a uniform policy. Therefore, in this view, $\beta$ can be interpreted as temperature.

    - **The perspective of DPO/RLHF**. As shown in Proposition 1, the optimal solution $\theta^\*$ to the DPO or the RLHF objective function, when the reference model follows a uniform distribution, achieves $\beta$-perfect alignment. Most interestingly, the $\beta$ is **precisely** the KL regularization strength parameter that appears in both the DPO and RLHF objective. **Thus, in this view, $\beta$ can be interpreted as the strength of KL regularization.**
  

• Suggested clarification presentation regarding formulating compound AI systems as DAGs, Figure 1 description, and typos will be incorporated in the camera-ready version.

We thank the reviewer once again for the encouraging feedback and for acknowledging the importance of compound AI systems alignment. We hope these clarifications and additions will strengthen the paper and more clearly convey its contributions to the study of alignment in compound AI systems.

We sincerely appreciate the reviewer’s insightful feedback and recognition of the compound AI systems alignment problem. To our knowledge, SysDPO is the first principled framework that jointly aligns the parameters of an entire compound AI system modelled as a DAG. For better fluency, we address the reviewer's comments and questions in the order of their appearance in the manuscript.

• Detailed explanation of $\beta$‑perfect alignment: We thank the reviewer for suggesting the perspective of the Boltzmann rational model. It is very interesting how $\beta$-perfect alignment can be precisely interpreted from many different perspectives. Details follow.
  • $\beta$-perfect alignment of generative models. Given an input $x$, consider two versions of the generated output $z^1$ and $z^2$. Consider a ground truth preference oracle $`pref`(z^1\succ z^2\mid x)\in (0, 1)$, which states how much $z^1$ is preferred over $z^2$. The model $\theta^\*$ is $\beta$-perfectly aligned to the preference oracle $`pref`(\cdot)$ if for any $x\in\mathcal{X}$ and $z^1, z^2 \in \mathcal{Z}$ such that $$ pref (z^1 \succ z^2 \mid x) = \frac{p _ {\theta^*} (z^1\mid x)^\beta}{p_{\theta^*}(z^1\mid x)^\beta+p_{\theta^*}(z^2\mid x)^\beta} \qquad\qquad{(\text{Definition 1})}

We sincerely thank the reviewer for the constructive and encouraging feedback. We will incorporate clarifications and revisions, as well as new experiments in the camera-ready version. Details regarding your comments follow.

• Insights of the theoretical analysis in Section 3.1 and how it extends standard DPO
  • Section 3.1 indeed revisits standard DPO and RLHF to show that they achieve $\beta$-perfect alignment. The purpose is to justify the definition of $\beta$-perfect alignment and establish a connection with existing results. Our theoretical contribution and new insights are discussed in Section 3.2: by factoring a directed acyclic graph (DAG) of components, we prove that SysDPO achieves the same β‑perfect alignment guarantee at the system level (under Assumption 1), even when the system has a non‑differentiable structure. To our knowledge, this is the first formal alignment guarantee in such compound AI system alignment.
  • Moreover, in the LLM+Diffusion model setting, deriving a tractable SysDPO loss function is non-trivial: diffusion models, unlike LLMs, do not directly output the log-likelihood of generated outputs. We address this through a tailored derivation (shown in Appendix D), extending the single-model Diffusion DPO (Wallace et al.). This is an important and nontrivial technical contribution not emphasized in the main text due to space limitations.
  • We will highlight these distinctions in the camera‑ready and move the key part of Appendix D into the main text.
• Theorem 1 seems to only show that the statement holds under Assumption 1 and not the necessity of this assumption.
  • We thank the reviewer for catching this. Yes, Assumption 1 is sufficient for proving $\beta$-perfect alignment, but it may not be necessary. In practice, diverse samples often suffice. We will clarify this point and revise the statement in the camera-ready version.
• Experiments beyond two components.
  • We acknowledge the importance of evaluation on large-scale multi-component systems. We include a new experiment on a three‑LLM system: LLM 1 and LLM 2 independently generate responses to the input question, and LLM 3 synthesizes these responses into a final answer. This setup extends our two-LLM system to work with more components. We train this system jointly with SysDPO and evaluate it using the WR‑Prompted metric. Without changing the hyperparameter, the aligned system achieved a 58 % win rate against its pre‑alignment version, showing promising potential for scalability. While broader experiments with many components are beyond our limited academic budget, we believe this result demonstrates that our framework provides a solid foundation for future scaling.
• Joint policy optimization vs separate policy optimization.
  • Thank you for raising this important distinction. Even in the two-component setting, we observe that separate DPO underperforms joint SysDPO as shown in Table 2. In this LLM+LLM setting, LLM 1 drafts an initial response, and LLM 2 refines it. For Separate-DPO, we train each model on the same task‑level dataset (Orca‑DPO‑Pairs) independently. In contrast, SysDPO aligns both models jointly, allowing updates that reflect their collaboration dynamics. This leads to a more coordinated and effective system. We will highlight this difference more explicitly in the camera-ready version, as it demonstrates the necessity of joint alignment.

We would like to once again thank the reviewer for their constructive comments and for acknowledging the importance of aligning compound AI systems. The clarifications provided above will strengthen the paper both theoretically and empirically.

References. Wallace et al. Diffusion Model Alignment Using Direct Preference Optimization.