Pre-Trained Policy Discriminators are General Reward Models

Thank you for carefully reviewing our manuscript and providing valuable suggestions. We have addressed each of the issues you raised and will incorporate revisions into our latest manuscript.

Weakness 1: Lack of contextualization

Weakness 1.1: The authors claim that using labeled preference pairs is limited by annotation difficulty, yet POEM also depends on human rankings during fine-tuning.

Answer: We apologize for any misunderstanding caused by our unclear wording. What we intended to convey is that other methods require large amounts of annotated preference pairs to train reward models, which makes them difficult to scale. As reported in our response to Reviewer 4MF9’s Question 2, existing reward modeling methods annotate a substantial number of preference pairs to achieve better performance. For example, training WorldPM-72B [1] and Skywork-Reward-8B [2] requires 15M and 40M labeled preference pairs, respectively, whereas fine-tuning our POEM-1.8B only requires 0.15M labeled data. The amount of labeled data needed to fine-tune the POEM model is much fewer than that required by existing SOTA reward models.

Furthermore, to further improve performance, these traditional reward models would require even more annotated preference pairs. However, manual annotation is resource-intensive, while automated annotation may introduce bias. In contrast, POEM exhibits strong generalization ability and achieves scalability through pre-training. It does not require the massive amount of annotated data needed by traditional reward modeling methods. This is the key point we wanted to express.

We will refine our statement and include a comparison of training data requirements between POEM and traditional reward modeling methods in the main text to help readers better understand our work.

Weakness 1.2: The paper criticizes Bradley-Terry Loss for assuming absolute preferences, yet POEM uses a similar method with three samples, showing recurring inconsistencies.

Answer: We apologize for any confusion caused by our unclear statements. We would like to clarify that we are not criticizing the form of the BT framework itself, but rather arguing against the way current reward modeling methods use the BT framework to model absolute preferences. Specifically, these methods utilize the BT loss to model what constitutes good or bad preferences. However, the preference data used for training may not be able to capture a wide range of preferences. Moreover, conflicting criteria may lead to contradictory partial orderings within the same training preference data.

In contrast, although POEM also employs the BT framework for training, it models relative difference. For example, in RLHF, relative difference can be understood as the gap between the training policy and the target policy, rather than making absolute judgments about individual policy.

Therefore, in the pre-training phase, POEM trains the model to distinguish between policies, while in the fine-tuning phase, it learns to align with human perception of the differences between two policies. Through this approach, POEM avoids modeling absolute preferences directly. As a result, POEM can scale its capabilities through pre-training without requiring extensive annotation of fine-tuning data.

We will refine our statements in our latest manuscript to more clearly articulate the limitations of traditional RMs.

Weakness 1.3: Needs better context and discussion of limitations.

Answer: Thank you for your valuable suggestions regarding our manuscript. In addition to addressing the two specific concerns you mentioned, we will thoroughly review and further refine our manuscript. We will do our best to ensure that our contributions are clearly and effectively communicated to the readers.

Weakness 2: I wonder if POEM’s success mainly comes from its reformulated loss. I may be missing something, so I’d appreciate the authors’ perspective.

Answer: Thank you for your valuable comments. As we mentioned in our response to Weakness 1.2, the success of POEM stems from a fundamental shift in its modeling objective (from absolute preference to relative difference).

Traditional reward modeling methods aim to enable RMs to learn absolute preferences. These methods require continuous annotation of preference data in an attempt to comprehensively cover human preferences, which is inherently difficult to scale. In contrast, POEM is designed to learn relative difference. During pre-training, POEM encourages the model to distinguish between different policies; during fine-tuning, it trains the model to align with human perception. Both stages share the same goal: modeling relative difference.

This shift in modeling objective makes unsupervised pre-training of the RM possible, similar to how pre-training is crucial for the success of LLMs. We can construct large-scale pre-training datasets for POEM without the need for manual annotation, continuously improving its performance. Our scaling experiments clearly show the scaling laws of POEM. Moreover, ablation studies further show that if the POEM model is only fine-tuned on annotated data without pre-training, its generalization ability drops significantly. This provides further evidence that pre-training brings strong generalization capabilities.

Weakness 3: No analysis on policy set sensitivity.

Answer: Thank you for your valuable suggestions. We agree that the choice of policies used to construct the pre-training dataset can affect the quality of the pre-training data, and consequently influence POEM’s capabilities. However, investigating this question requires substantial computational resources. As reported in Appendix C, training POEM-1.8B consumed significant computational resources: we used 320 NVIDIA H20 GPUs for approximately 57 hours. Therefore, due to budget constraints, we finally utilized all available policies (184 models) in practice.

Fortunately, even without carefully selecting the policies for pre-training, POEM-1.8B demonstrated remarkable performance, showing strong effectiveness and generalization in both preference prediction evaluation and RLHF, significantly outperforming all SOTA baselines.

In the future, we will ensure to invest more computational resources to better understand the impact of policy set selection on performance. We also plan to train larger models to further advance POEM’s performance.

Weakness 4: The paper justifies pretraining but not fine-tuning in POEM’s two-stage training. An ablation study on fine-tuning would be valuable.

Answer: We conduct an ablation study by evaluating the preference prediction performance of the pre-trained-only model, as shown in the table below.

We observe that, without alignment to human judgments, POEM-1.8B fails to achieve meaningful scores on preference evaluation. This is reasonable, as only after POEM’s judgments are aligned with human judgments can it effectively leverage the broad knowledge acquired during pre-training to achieve better performance on downstream tasks. This is similar to LLMs, where LLMs need to be fine-tuned with a small amount of labeled data to align with human instructions. In essence, fine-tuning enables POEM to distinguish between policies from a human perspective.

Combined with the ablation study in our manuscript, which evaluates POEM without pre-training, it is clear that both stages of POEM play an irreplaceable role.

Weakness 5: Introduction and contributions are vague and unclear.

Answer: Thank you for your valuable suggestions. We will add more details about our method in the introduction to help readers gain a better understanding of our approach early on.

Question1-4: Some typo, notation, and expression issues

Answer: Thank you for your valuable comments! We have addressed the following points:

(1) Notation (Line 123): We confirm that $\tau$ should refer to a single trajectory, not a set. We will revise the notation for clarity.

(2) Equation Citations: Equation 1 follows the RL objective in Ouyang et al. [3], and Equation 2 is based on the closed-form solution from Rafailov et al. [4]. We will add the appropriate citations.

(3) Complementary Criterion (Line 140): Good catch. We will revise the phrasing and have confirmed that this does not impact the validity of our subsequent arguments.

(4) Notation in Eq. 6: $\sigma$ denotes the sigmoid function. We will clarify this explicitly in the revised manuscript.

Moreover, we will carefully review the entire paper to ensure that all concepts are clearly explained, further helping all readers to better understand our work.

Finally, we would like to sincerely thank the reviewer for all the valuable suggestions. We will make sure to incorporate all of the reviewer’s suggestions into our latest manuscript. If you have any further comments, please do let us know, and we will do our best to address them.

References:

[1] Wang, B. et al. (2025). WorldPM: Scaling Human Preference Modeling. arXiv.

[2] Liu, C. Y. et al. (2025). Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy. arXiv.

[3] Ouyang, L. et al. (2022). Training language models to follow instructions with human feedback. NeurIPS, 35, 27730-44.

[4] Rafailov, R. et al. (2023). Direct preference optimization: Your language model is secretly a reward model. NeurIPS, 36, 53728-41.

Thank you very much for your recognition of our work and for providing valuable suggestions. We have addressed each of the issues you raised and will incorporate the corresponding revisions into our latest manuscript.

Weakness 1: Dependence on reference trajectories from top-performing LLMs.

Answer: Thank you very much for your valuable comments. We provide a comprehensive discussion on why this limitation may not be a significant issue for several reasons, as detailed in our response to Reviewer cmE2’s Weakness 1. We hope this further discussion addresses your concern. If you have any further questions or would like to discuss in more detail, please feel free to contact me.

Weakness 2 & Question 1: The paper claims that “trajectories derived from other prompts may also effectively characterize policy behavior" just as well as using the reference trajectory for the current prompt (Section 5) but does not include quantitative evidence or error bars demonstrating robustness to reference quality. Could you provide empirical results—e.g., a plot of POEM performance when using the trajectory from the current prompt as reference versus using a trajectory of the same reference policy but from a different prompt—to substantiate the claim you make in Section 5?

Answer: Thank you for your insightful question. We would like to clarify that our discussion on this topic is presented in Section 5 (Limitations and Future Work), where we mainly aim to share our latest observations from the case study. This does not include systematic quantitative experiments, and we would leave them as future work. We appreciate the opportunity to further elaborate here.

The motivation behind using trajectories from different prompts is to alleviate POEM’s reliance on prompt-specific references. POEM is designed to assign reward values based on the consistency between two different policies. In our preliminary exploration, we observed that the reference and the trajectory can correspond to different prompts, as long as they share commonalities.

Specifically, POEM was originally intended to score based on (Prompt, Reference, Trajectory). However, we found that it can also score based on (Prompt A, Reference, Prompt B, Trajectory), where "Reference" represents Policy A's answer to Prompt A, and "Trajectory" represents Policy B's answer to Prompt B. Formally, the input format changes from
<|bos|> Prompt <|split_token|> Reference <|split_token|> Trajectory <|eos|>
to
<|bos|> Prompt A <|split_token|> Reference <|split_token|> Prompt B <|split_token|> Trajectory <|eos|>.

We illustrate this with the following case study:

Prompt A: "All birds have wings. Tweety is a bird. Question: Does Tweety have wings?"

Reference: "Yes. Tweety has wings."

Prompt B: "All teachers work at schools. Mr. Smith is a teacher. Question: What can we conclude?"

Trajectory 1: "Mr. Smith works at a school." # Correct
=> Reward: -9.0390625

Trajectory 2: "People who work at schools are teachers." # Incorrect
=> Reward: -11.015625

In this case, both Prompt A and Prompt B are classic deductive syllogisms, each requiring reasoning from two premises (one general and one specific) to reach a necessary conclusion. Although the question types differ (Prompt A is yes/no, while Prompt B is open-ended), the underlying logic is similar. For Prompt B, Trajectory 1 provides the correct conclusion, whereas Trajectory 2 incorrectly reverses the premise.

Results show that, when given Prompt A and its reference, POEM can assign a higher reward to the correct answer to Prompt B. This suggests that POEM has the potential to generalize scoring across prompts by referring to similar question-answer pairs. Such a capability could help reduce the annotation burden of generating a reference for every individual prompt.

We will update Section 5 (Limitations and Future Work) to clarify the current scope of our findings. In future work, we plan to further investigate this finding and systematically evaluate POEM’s performance when using references and trajectories from the same versus different prompts.

Question 2: Was POEM initialised from an existing pretrained LLM (and if so, which architecture and checkpoint), or trained from scratch?

Answer: The pre-training phase of POEM is initialized from an existing pretrained LLM. Specifically, POEM is pre-trained starting from the InternLM2 series InternLM2.5-1.8b-chat model [1], as presented in Appendix C.1.2 (Implementations, Line 837) of our manuscript, due to space limitations in the main text. We will move this information to the main text to provide readers with a clearer understanding of our training setup.

Finally, we sincerely thank the reviewer for their recognition and for all the valuable suggestions provided for our manuscript. We will make sure to incorporate all of the suggestions into the next version of our manuscript. If you have any further comments, please do let us know, and we will do our best to address them.

References:

[1] Cai, Zheng, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen et al. "InternLM2 Technical Report." CoRR (2024).

Thank you very much for providing valuable suggestions. We have addressed each of the issues you raised and will make revisions to our latest manuscript.

Weakness 1: The most significant weakness is that the reward model requires a reference during reward generation.

Answer: Thank you for raising this concern. Compared to traditional RM methods, POEM does require the introduction of additional references, which we have discussed in the limitations section. We hope the following further discussion will address your concern:

(1) In our RL experiments, POEM significantly outperforms conventional reward models. Given POEM’s strong performance, generalization, and scalability, the requirement of references is not expected to substantially limit its broader adoption and future development. One supporting example is Reinforcement Learning from Verifiable Reward (RLVR), a widely adopted post-training paradigm for verifiable tasks. In RLVR, each prompt also requires a corresponding reference answer, enabling the verifier to assess the correctness of the trajectory. For example, in mathematical reasoning tasks, it is common practice to use rules to compare the generated solution with the reference answer for reward assignment. RLVR has become popular precisely because its reward signals are sufficiently reliable, and the need for references has not hindered its widespread adoption. POEM can be regarded as a general-domain verifier. Experimental results show that POEM has generalization and scalability, which provides stable and accurate reward signals in RL, significantly outperforming SOTA reward model baselines.

(2) The need for references can be addressed through open-source LLM instruction tuning datasets and data synthesis. Specifically, the reference requirement for POEM is consistent with that for instruction tuning in LLMs, where each prompt also requires a reference. There are now many high-quality, open-source instruction tuning datasets that can serve as references for POEM. For references that are difficult to annotate, they can be synthesized using more powerful LLMs. In fact, the references used in our RLHF experiments were also synthesized (as indicated in Line 906). The results still show that POEM can achieve significantly better performance than all SOTA RM baselines, even without requiring a perfect reference.

In summary, although POEM requires references during reward generation, this limitation may not be a major concern due to the following factors: 1. the need for references is consistent with the verifier in the widely adopted RLVR paradigm, and POEM can be viewed as extending the verifier approach from verifable tasks to more general domains; 2. its data requirements are aligned with those for instruction tuning in LLMs; and 3. our experiments demonstrate that synthesizing references is feasible. Therefore, the need for references may not be a major concern, especially given the significant advantages POEM offers over traditional RMs.

Weakness 2: Assuming access to high-quality references makes it difficult to consider the comparison with baselines truly fair and poses practical limitations for real-world deployment.

Answer: We would like to clarify that we have tried our best to ensure a fair comparison with existing SOTA reward model baselines. As mentioned in the Evaluation Setup section (line 237), to ensure fairness, we evaluate these baselines using two approaches, i.e., scoring with and without the reference. For each baseline, we report the best results across these two settings. Additionally, as discussed in our response to Weakness 1 and in the manuscript (line 906), references can be synthesized from top-performing LLMs.

The results show that, even when using synthesized references as input, POEM-1.8B outperforms these SOTA baselines in all settings, while using significantly fewer parameters. POEM-1.8B also exhibits stronger generalization capabilities. We hope these clarifications address your concerns.

Question 1: The authors mention that they report the best results between reference-free and reference-included evaluations(lines237-238), but it would be more informative to separate the two and compare performance explicitly based on the presence or absence of the reference.

Answer: Following your suggestion, I have reported the performance of the two SOTA baselines in these two settings on the preference prediction evaluation, as shown in the table below.

The experimental results indicate that, for most tasks, providing a reference actually leads to decreased preference prediction performance for these traditional reward model baselines. This may be because these traditional reward models are optimized to model absolute preferences (i.e., determining what is good or bad), rather than relative difference, which focuses on comparing the quality of two policies.

Question 2: The impact of the pre-training stage on performance does not appear to be very large. As shown in Table 9, the average difference between POEM and the model w/o pre-training is only about 1.4 points. It would be helpful if the benefits of the pre-training stage were explained in more depth.

Answer: Thank you for your valuable question. Overall, pre-training enables POEM to achieve a strong generalization ability in RL practice. In fact, Table 9 presents a comparison between POEM with and without the pre-training phase in preference prediction evaluation. While preference prediction is the most commonly used and convenient setting for validating reward model effectiveness, it is not considered the “gold standard” for evaluating the true effectiveness and generalization of reward models, in fact, the performance of reward models in RLHF provides a more reliable evaluation method [1].

In Table 10 of our manuscript, we compare the effectiveness of POEM in RLHF with and without the pre-training phase. The results show that pre-trained POEM outperforms its non-pre-trained counterpart on nearly all tasks (19 out of 20), providing strong evidence that pre-training significantly enhances the generalization ability. Moreover, we explored the scaling laws during the pre-training phase, as shown in Figure 4. The results indicate that POEM has the potential to achieve even stronger performance through scaling.

In summary, POEM leverages unsupervised pre-training on large-scale data to learn to distinguish differences between policies. The pre-training phase is indispensable, serving as the foundation for POEM’s strong generalization ability and scalability.

Question 3: In scenarios where a reference is available, alternative methods could compute rewards based on similarity to the reference (e.g., embedding similarity). How would such approaches perform compared to POEM, and what are the fundamental differences between them?

Answer: We conduct experiments comparing POEM-1.8B with the SOTA embedding model on preference prediction evaluation. For the baseline, we calculate the similarity between the embeddings of the reference and the trajectory as the reward value. The experimental results are shown in the table in response to Question 1. The results show that, even when using OpenAI’s most powerful embedding model, our method significantly outperforms it in preference prediction.

There is a fundamental difference between our method and these similarity-based approaches. Although POEM also takes two samples as input, it measures the consistency between two policies represented by these two samples, rather than just their surface-level similarity. By focusing on this objective, POEM can effectively evaluate the consistency between the training policy and the target policy in RL, assigning more meaningful rewards.

Paper Formatting Concerns: Some of the figures and tables discussed in the main text—such as Table 9, 10, and 11—are placed in the appendix.

Answer: Thank you for your suggestions! In fact, Table 1 in the main text is a simplified version of Tables 9, 10, and 11, which aggregate the benchmarks according to their evaluation capabilities. We will add this explanation to the caption of Table 1 to help readers better understand the relationship among these tables. Additionally, if our work is accepted, we will move these tables into the main text to further improve the clarity of our paper.

Finally, we sincerely thank the reviewer for all the valuable suggestions provided for our manuscript. We will carefully revise our manuscript and incorporate these experiments into our latest version. If you have any further questions or suggestions, please feel free to reach out to us.

References:

[1] Zhou, E., Zheng, G., Wang, B., Xi, Z., Dou, S., Bao, R., ... & Huang, X. RMB: Comprehensively benchmarking reward models in LLM alignment. In The Thirteenth International Conference on Learning Representations (ICLR 2025).

Thank you for your response. We are glad that our rebuttal has resolved many of your questions, and we sincerely appreciate your recognition of our work and your decision to raise the score.

We agree that achieving an entirely fair comparison is inherently challenging, as POEM introduces a novel and scalable reward modeling paradigm that fundamentally differs from traditional approaches. Nevertheless, we have sought to ensure fair and rigorous comparisons with SOTA reward model baselines in our experimental setup.

Thank you again for your valuable comments and for acknowledging our efforts!

Thank you very much for your recognition of our work and for providing valuable suggestions. We have addressed each of the issues you raised and will incorporate the corresponding revisions into our latest manuscript.

Weakness 1: Preliminaries: The paper would benefit from a dedicated section summarizing the setup and notation, especially for readers less familiar with reward modeling in LLM alignment.

Answer: Thank you for your valuable suggestions. At the beginning of Section 3 (Methods), we will add an introduction to the experimental setup, notation, and conventional reward modeling methods to help readers better understand alignment and the contributions of our work.

Weakness 2: Related Work: The method bears strong conceptual similarity to prior works such as Generative Adversarial Imitation Learning and Discriminative Reward Co-Training. A discussion on how this work extends or differentiates itself from these would help contextualize its contributions.

Answer: Thank you for your insightful suggestion. We have carefully reviewed the two papers provided by the reviewer. Both GAIL [1] and DIRECT [2] train a discriminator that distinguishes between trajectories generated by the current policy and those from high-quality trajectories, using the discriminator's output as a reward signal. In GAIL, expert demonstrations are used as high-quality trajectories, whereas DIRECT stores historically high-return action sequences as high-quality trajectories. These works primarily focus on enabling the training policy to perform similarly to an expert policy within a single task. In contrast, our work investigates whether reward models in LLM post-training can adopt a pre-training paradigm and exhibit strong generalization capabilities, similar to LLMs.

Specifically, we explore how to construct unsupervised data to pre-train a policy discriminator, thereby redefining the reward function in LLMs. We further show its scaling laws and demonstrate its generalization and scalability through extensive experiments. Moreover, POEM can be fine-tuned with a small amount of labeled data to better adapt to downstream tasks, analogous to SFT in LLMs.

In summary, our work differs significantly from previous studies in terms of training methodology, data synthesis paradigm, model generalization and scalability, as well as application scenarios. We will incorporate these points into our latest manuscript. Furthermore, we will thoroughly review related works that may share conceptual similarities with POEM and further expand Section 2 (Related Work) to help readers better understand our contributions.

Question 1: Could the authors clarify the intuition behind the fine-tuning loss in Eq. 7? Specifically, what motivates replacing $(A_1, A_2 | B_1, B_2)$ with $AC | AB$? A brief intuitive explanation would enhance understanding.

Answer: POEM consists of two stages, i.e., the pre-training phase and the supervised fine-tuning phase. The goal of the first stage is to pre-train the reward model to distinguish between different policies. However, at this stage, the reward model is not yet aligned with human judgment; specifically, it does not capture the extent of differences between policy distributions as perceived by humans, nor does it reflect whether humans consider two policies to be different. Therefore, it is necessary to further fine-tune the reward model on a small amount of labeled data to achieve alignment with human judgment.

Another intuitive explanation can be drawn from the training paradigm of LLMs. Although pre-trained LLMs acquire extensive knowledge, they are not directly suitable for downstream tasks. Instead, they require fine-tuning with a small amount of labeled data to learn to follow human instructions and align their language behavior with human expectations. We will refine our manuscript and incorporate these intuitive explanations to help readers better understand the purpose of the fine-tuning stage in POEM.

Question 2: What are the compute and data requirements for training the discriminator at scale compared to other alignment strategies (e.g., current preference models)?

Answer: We compared our method with SOTA baselines in terms of compute and data requirements for training, as summarized in the table below. Missing values in the table indicate that the corresponding numbers were not disclosed by the respective methods.

Several key conclusions can be drawn from this table:

(1) Our method introduces a pre-training phase, which requires additional compute resources. However, this phase is unsupervised and does not require labeled data, similar to the pre-training stage of LLMs. (2) The amount of labeled data required by our method is significantly lower than that of traditional reward modeling approaches. Specifically, we only need 150K labeled samples for supervised fine-tuning, whereas other methods typically require millions of labeled samples, which substantially increases annotation costs. (3) Moreover, our model has only 1.8B parameters, making it much more accessible for users to fine-tune POEM-1.8B compared to these larger models.

In summary, although POEM consumes more compute during the pre-training phase, users can fine-tune POEM-1.8B with significantly less computational and annotation resources to obtain a more powerful reward model compared to other baselines.

Furthermore, after the anonymity period ends, we will open-source all our models, including the pre-trained model, SFT model, and training framework, to minimize user adoption costs. Users can directly utilize the SFT model or further fine-tune the reward model with a small amount of domain-specific data to achieve enhanced performance.

Question 3: How does the scalability presented in 4.4 compare to the considered baseline approaches?

Answer: Our method shows significantly better scalability compared to SOTA reward model baselines. As discussed in Section 1 (Introduction, line 26) and Section 2 (Related Work, line 90) of our manuscript, these baseline methods require the continuous annotation of a large number of preference pairs to improve performance. Consequently, they lack the scalability. In contrast, the pre-training phase of POEM is unsupervised and does not require labeled data, which ensures excellent scalability. During the fine-tuning phase, we only use a substantially smaller amount of labeled data than baseline methods to fine-tune the pre-trained model, yet still achieve stronger performance (as shown in experiments of our manuscript).

Moreover, our scaling laws experiments (Figure 4) show a predictable decrease in validation loss as model size or computational resources increase. Furthermore, Tables 1, 5, 6, 7, and 8 demonstrate that POEM exhibits significantly better generalization and effectiveness compared to other SOTA baselines.

In summary, these experimental results clearly indicate that our method offers substantially superior scalability.

Finally, we sincerely thank the reviewer for their recognition and for all the valuable suggestions provided for our manuscript. We will make sure to incorporate all of the suggestions into the next version of our manuscript. If you have any further comments, please do let us know, and we will do our best to address them.

References:

[1] Ho, Jonathan, and Stefano Ermon. "Generative adversarial imitation learning." Advances in neural information processing systems 29 (2016).

[2] Altmann, P., Ritz, F., Zorn, M. et al. Discriminative reward co-training. Neural Comput & Applic (2024). https://doi.org/10.1007/s00521-024-10512-8

[3] Wang, B., Lin, R., Lu, K., Yu, L., Zhang, Z., Huang, F., ... & Lin, J. (2025). WorldPM: Scaling Human Preference Modeling. arXiv preprint arXiv:2505.10527.

[4] Liu, C. Y., Zeng, L., Xiao, Y., He, J., Liu, J., Wang, C., ... & Zhou, Y. (2025). Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy. arXiv preprint arXiv:2507.01352.