AI-Slop to AI-Polish? Aligning Language Models through Edit-Based Writing Rewards and Test-time computation

Benchmark Contribution The WQ benchmark includes one newly collected dataset (Synthetic Mirror), two datasets requiring extensive processing to create pairwise preferences (Art or Artifice and LAMP-test), and one heavily filtered to focus on creative writing quality (LM Arena). Only Style Mimic was used without major modifications. This comprehensive processing work was necessary as no existing benchmark focuses specifically on writing quality assessment. Due to length limitations, these details were included in the Appendix (Appendix D1) but we will write a summary of the processing we performed in the main portion of the paper to highlight our contribution. The paper also includes expert annotations from 9 professionals (detailed in Section 6) that will be released publicly upon acceptance. Finally consolidating multiple datasets to create a benchmark has long standing evidence in NLP/AI. For example GLUE, SUPERGLUE which fueled a lot of early research on language understanding.

Modeling , Design and Experimental Contribution: We highlight several innovative contributions:

(i) using implicit preferences from edits for training reward models that perform well on benchmark datasets helps dealing with noise and inherent disagreement thats prevalent in collecting preference data for RLHF

(ii) Our PR models uniquely combine scalar rewards and pairwise preferences into a single model, which empirically outperforms separate models when both signal types are available;

(iii) We demonstrate small encoder-only models can compete with larger generative models;

(iv) We extend rationale-based training benefits for LLM as a judge especially for subjective open-ended tasks like writing quality evaluation

(v) Our test-time compute editing pipeline for over-generation and candidate selection via WRQM extends TTC benefits to tasks with non-verifiable rewards, proving its value beyond traditional math/reasoning tasks.

Practical Utility Contribution: : See here

We hope that our points highlight some of the contributions of our work. We look forward to engaging further with you on the listed contributions. If our response has provided the clarity you were seeking, we would humbly request that you consider adjusting your rating accordingly.

Thank you so much for the very positive evaluation of our paper.See our responses inline

Reasons to Reject

Writing Quality Definition: We agree with the reviewer that writing quality is challenging to define. Our definition of writing quality was based on the taxonomy introduced by Chakrabarty et al. (2024b), which itself is based on edits performed by professional writers as they improve the writing quality of text. We will specify this clearly in the updated version. A good quality written text should avoid clichés, purple prose (overly ornamental language), unnecessary exposition, poor sentence structure (very long run-on sentences), awkward words and phrases, and lack of specificity and detail. In short, it’s a combination of fluency, coherence, clarity, and style.

Efficiency: We acknowledge that test-time computation is expensive, but it’s a newer and widely used approach for improving LLM responses. To mitigate cost, we chose a small and efficient mBERT model for the reward model, which lowers computation and improves latency to some degree. Developing efficient methods that leverage WQRM to generate high-quality writing responses is an exciting avenue for future work.

Answers to Reviewer

1. Space constraints: We were constrained by space. In the revised draft, we will highlight the crucial details from Chakrabarty et al. (2024b) that may have been obfuscated.

2. Joint optimization of regression & classification: We applied a weight of 1.0 to the classification loss and 0.25 to the regression loss, chosen so that at initialization both losses have roughly the same magnitude. We did not exhaustively explore this hyperparameter’s downstream impact and will add these details to the training description.

3. Rationale-based improvement: We observed that adding natural-language rationales boosted performance most on the harder WQ subsets (Art or Artifice, Style Mimic). Table 6 in Appendix 6 provides examples of these rationales. Our design cites specific paragraph excerpts to justify judgments, making the process transparent: if a rationale cites clichés or other flaws, the model learns to penalize those traits. This aligns with recent work on “thinking before judging” [1, 2].

References

[1] Huang et al., Think-J: Learning to Think for Generative LLM-as-a-Judge, 2025.
[2] Chen et al., RM-R1: Reward Modeling as Reasoning, 2025.

First of all, we want to thank you for highlighting the positive aspects of our paper and asking great followup details. We are glad that you found value in our analysis in Section 6. Please find our responses inline.

Addressing questions around methodological details:

1. Regarding tie resolution for R-mode models: We performed an analysis on the MBERT-WQRM-R model. We consider a tie if a pair's score differs by less than 0.001 (remember the scores are on a scale of 0-10). Only 4 of 4,729 pairs in the WQ test set were ties, meaning different tie-breaking methods would change results by at most 0.1%. We'll update our tie-breaking procedure description to note these cases are rare.

2. PR -> P versus PR -> R: Regarding the mode of use of the PR-trained models, we indeed experimented with both P-mode and R-mode, and found that R-mode use outperforms P-mode. We tested both P-mode and R-mode, finding R-mode superior. For MBERT-WQRM-PR, R-mode achieves 74.3% on WQ (reported in paper) versus 66.3% in P-mode (unreported)—an 8-point drop. We will update the discussion of the PR models to add this important nuance to the findings: models trained jointly on pairwise and reward-based datasets perform better when evaluated in R-mode in our experiments.

3. Regarding the reversal of relationship between the P-trained and R-trained models between the MBert and Llama families: Great observation !! Since Llama is trained on whole number rewards only (1-10), the causal language modeling loss effectively treats this as a 10-way classification task - it's learning to predict which of the 10 possible reward tokens comes next. Meanwhile, MBert uses custom losses (pairwise classification for P, MSE for R) and can handle continuous values and is explicitly optimized for regression, allowing it to better capture the underlying reward structure even when trained on the same whole number targets. This leads the MBERT-R models to outperform MBert-P models, whereas the reverse is true for the LLama models, as they are not able to properly take advantage of the R-based data. Future work could investigate tuning Llama models with custom losses (MSE), which we hypothesize would lead to improved LLama-R models. We will add this discussion to the findings section of our results.

4. Regarding your question about the distribution of scores for the different portions of the WQ benchmark to get a sense of level of difficulty. This is a great idea, however as you point out only one of the five datasets in the benchmark have scored data. However, based on your suggestion we performed an additional analysis, leveraging our WQRM-PR model to "annotate" the samples with scores, and observe the distribution. See the Table below.

Table 1: Average WQRM scores for “worse” vs. “better” writing examples across different WQ datasets.

Analysis. A few points are noteworthy:

• (4a) The average gap is directly proportional to model performance on the benchmark. The Synthetic Mirror dataset has the largest gap according to WQRM-PR (2.4 on average), and many models reach near-perfect accuracy (98%+) on it. In contrast, the gap is very small on Style Mimic (0.12) and LMArena (0.02), which aligns with many models performing at or only slightly above chance on those splits.

• (4b) The absolute scores for the “worse” and “better” samples reflect their origins. Style Mimic is the only dataset with Human–Human comparisons (both written by professionals), and both samples score highly (7.57 vs. 7.69). LMArena also shows a small gap, but at lower overall quality levels (5.99 vs. 6.02).

• (4c) Together, (4a) and (4b) show that the WQ benchmark spans both high-gap (easy) and low-gap (hard) settings. Low-gap cases can be two low-scoring AI generations or two high-scoring human texts, confirming the benchmark’s breadth which is one of our key design goals.

• (4d) We want to say WQRM-PR is only a proxy for true quality scores (expert annotated labels would be cost-prohibitive). Still, the trends match our expectations & validates both the calibration of WQRM-PR & the diversity of the WQ benchmark. We will include this analysis in the Findings section (moving the table to the Appendix for space).

If our response has provided the clarity you were seeking, we would humbly request that you consider adjusting your rating accordingly.

First of all, we want to thank you for highlighting the positive aspects of our paper. Please find our responses inline.

Evidence for Generality: The WQ benchmark was constructed specifically with generalization in mind. Our benchmark consists of AI-AI, Human-AI, Human-Human and AI-HumanAI pairs, covering all possible combinations for quality judgment. Our models were trained on AI and HumanAI paragraphs but can still transfer to purely AI-written or purely human-written paragraphs, showing signs of generalization. Not only that, our models can also do writing-quality judgments for long form / long context (something it wasn’t trained on). The paragraph length for Art or Artifice is 5–6× longer than the average paragraph length in training. See also (Some Additional Experiments highlighting Practical Utility of WQRM) for evidence of generalization.

Human Evaluation sample size : We conducted the reward-model calibration study with 9 experts (which adds credibility to our approach). For tasks such as writing, it’s often better to have a smaller group of highly qualified expert annotators compared to a larger pool of non-experts. This has also been the trend in several human-centered studies focusing on writing ¹².

Domain Coverage: Our test data for WQ consists of literary fiction, creative nonfiction (cooking, travel, internet advice) and our additional human evaluation (100 prompts: literary fiction, news writing, marketing). We think this constitutes a broad representation of domains and genres. Unfortunately, covering every writing task would be cost-prohibitive, so we selected domains that are common and popular. Thank you for sharing arXivEdits; it’s indeed relevant and we will absolutely add a citation. Although we considered scientific writing, early rewriting experiments often hallucinated or produced inaccuracies, so we deprioritized it. We’ll note this in our Limitations.

Comparison with Commercial Tools / Self Refinement: We’re not aware of any commercial tool that judges writing quality. Our focus is on learning robust rewards for quality rather than self-refinement. Indeed, Section 5.1 uses targeted self-refinement but without knowing what to refine (or if intermediate refinements truly improve quality), you risk reward hacking ³.

Practical Utility Contribution: See Here

Thank you for acknowledging our response. If we have addressed your concerns, we would humbly request that you consider adjusting your rating accordingly. We are happy to engage should have any further comments.