SAG: Style-Aligned Generation via Language Model Collaboration

1. In the second paragraph of the Introduction part, you'd better provide 2-3 specific examples of recent, influential papers for each approach (training-free and training-based), in order to inform readers about recent research in this field.

2. Too many empty lines in Introduction Section (Line 80 - 84), which makes the paper looks untidy.

3. In Line 86, you mention the difficulties of balancing style-following capabilities with effective content generation. Do you mean the challenge of balancing stylic text generation capabilities and effective model training? There should be a more accurate description here.

4. In the third paragraph of the Introduction part, I think the way of how your framework handles those problems should be emphasized accordingly, in order to clearly show your motivations. Such as describing as "to alleviate catastrophic forgetting of directly fine-tuning LLMs, our framework xxx. To improve generation capabilities of SLMs, our framework xxx."

5. In the Introduction part, people may have different definitions of small language models (SLM). How small a model is that can be named as SLM? Maybe some model names can be listed as examples of the SLM here.

6. In Section 3.1, there should be a formal definition of the style imitation task. You could include key components such as input variables (e.g., content summary, style reference), output variables (e.g., stylized content), and the objective function or evaluation criteria

7. Evaluation Benchmarks of Section 4.1, some recent reference-based scores rather than ROUGE and BLEU should be considered, such as BLEURT (Sellam et al., ACL 2020) and BERT-Score (Tianyi et al., ICLR 2020). On the other hand, you'd better explain why ROUGE and BLEU might be more suitable for evaluating style imitation tasks.

8. In Section 4.2, since there are quite a mount of inferior results in the performance comparison, I think it's better to specifically analyze the reasons of leading to the performance reduction here.

1. The collaborative training framework introduced by the authors is based on using LLM to generate data for training SLM. However, whether using prompting to guide LLM or the two-stage training of SLM through SFT-DPO is quite common, and this approach seems to lack technical innovation.

2. Since the task of this paper focuses on style, using BLEU and ROUGE as evaluation metrics appears somewhat inadequate. It would be more appropriate to introduce metrics that assess the style, or consider incorporating human evaluation.

3. Although the authors conducted extensive experiments to demonstrate the effectiveness of the framework in improving both SLM and LLM independently, they did not compare their method with other Style Transfer approaches, which makes the framework's effectiveness less convincing.

The most significant issue in the paper is the inconsistency in the reported results across different tables.

• Table 3 vs. Table 4: Taking the results of the 1.8B models as an example, the Style Consistency score for Qwen in Table 3 is identical to the score for Gemini-Pro in Table 4, and vice versa. I believe one of the tables has swapped the results for these two models, which undermines the reliability of the findings.
• Table 6 vs. Table 7: A more serious issue arises when comparing results under different LLMs in Table 6 and Table 7. According to line 371, Table 6 uses Qwen-72B-Chat as the LLM for reference generation, where the Rouge-1 score for SAG is 36.19. However, in Table 7, where GPT-4 is the LLM (line 454), the SAG model achieves the same Rouge-1 score of 36.19, and all other metrics are identical. This suggests a significant error in reporting, as these results should not be identical under different LLMs.

Due to these major errors in the results, I think the paper’s claims are unreliable. The authors need to thoroughly check and correct the results in the tables to ensure they align with the actual experimental findings.

Beyond these issues, there are a few additional concerns:

1. Inconsistent experimental settings. The experimental setups vary across different sections of the paper, making it difficult to compare results between experiments. For example, the main experiments in Table 4 use Qwen 1.8B and Qwen 7B as the SLMs, but Table 5 and 6 do not clearly specify the SLM used, while Table 7 employs Qwen 14B as the SLM. These inconsistencies in the experimental settings cast doubt on whether the conclusions drawn from the analysis experiments can be applied to the main experiments. The authors should standardize the experimental settings across all experiments unless there is a specific reason to vary them, which should be clearly explained.
2. Lack of comparison with existing methods. The paper evaluates its performance solely on its self-created dataset and does not provide comparisons with existing methods. Given the number of well-established datasets and methods for style transfer and role-playing, the lack of comparisons makes it difficult to assess the effectiveness of the proposed approach. Including comparisons with existing approaches would strengthen the paper’s contribution and validate its claims.
3. Unclear implementation of baselines. The paper lacks details on how the baseline models were tested in Table1, including whether zero-shot, few-shot, or other techniques were used. Providing this information, along with example prompts in the appendix, would help readers evaluate the baselines properly.
4. Lastly, the paper claims in line 224 that “The detailed prompts and examples are provided in Appendix A,” but only examples are included, with no prompts.

• The paper provides strong motivation for the collaborative training framework; however, it lacks clarity on the necessity of introducing a new task to demonstrate its effectiveness. Specifically, the proposed task—imitating a specific style while generating content from a summary—appears somewhat impractical and lacks clear theoretical or practical justification. The value of this task remains ambiguous, as the paper does not convincingly demonstrate its relevance or potential applications in real-world scenarios.
• The fatal weakness of this paper is its reliance on ROUGE and BLEU scores within the NoteBench benchmark to assess style consistency. While these metrics are frequently used for measuring lexical overlap, they fall short in capturing stylistic nuances such as tone, vocabulary, and structure, which are central to the style imitation task proposed by the paper. Let alone that they're indeed not reliable evaluation metrics for NLG tasks.
• btw. This is the first time I see someone use metrics like BLEU to evaluate the style. Too often, they were used to measure content preservation in style transfer tasks.
• Similar problems apply to the evaluation of hallucination. The paper relies on LLM-based annotation for hallucination detection without establishing the reliability of this approach. Emerging research, such as the hallucination detection task in Semeval 2024, suggests that LLMs are not yet fully dependable for hallucination annotation (see: https://helsinki-nlp.github.io/shroom/2024).