We appreciate the reviewer's insightful feedback. We provide a detailed response below to address the concerns and questions raised by the reviewer.

W1. the methodology is straightforward and standard

Our goal is not to propose yet another alignment method but to introduce the insight that alignment can be driven by constraints, specifically tailored for customized use cases. The research question we address is: If human annotation is unavailable and models cannot perform well on a task, where should the supervision signals come from for customized alignment? Our primary focus is to conceptualize and validate this across various scenarios. We are the first to bridge LLM alignment with constraint-driven learning. To support this, we formally categorize existing constraints into three types, demonstrate their integration into the alignment process, and showcase the effectiveness and transferability of representative constraints.

W2. The topic is arguably not one of the most fanciest

Our work addresses a critical and underexplored problem in LM customization:

Assume one has an off-the-shelf LM at hand, with no access to how this LM was trained by the provider (training data, RM model, etc.). However, this LM may not meet one’s needs in terms of constraint satisfaction, even if the constraint requirement is explicitly added to the prompts. The challenge is how to improve the LM's constraint adherence as cost-effectively as possible.

• Approach 1: Post-processing — use the LM as-is and perform some post-editing to satisfy constraints (Inference with Constraints baseline in our paper).
• Approach 2: Collect annotations for the task at hand and finetune the LM on this data (Finetuning as reference in our paper).

This paper argues that both approaches are not ideal because Approach #1 underperforms, and Approach #2 is too costly. Thus, our approach focuses on the following problem: if the main issue with the LM is constraint-following, which can often be checked automatically, how do we automatically distill this knowledge into the LM?

We believe our work offers a promising and impactful contribution to advancing the customized alignment of LLMs.

W3. Could one have added more baselines, e.g., constrained decoding?

Constrained decoding is orthogonal to our work and is considered one type of inference with constraints in our paper. (In other words, constrained decoding is for inference, while ACT is for training.) We have already presented the results of such methods. We show that ACT and inference with constraints are complementary. By combining our method with inference with constraints, one can achieve better performance. As noted in Footnote 3, we have tested various inference-with-constraints methods and observed no significant performance difference. For consistency, we report the results of using inference with constraints derived from constraint verifiers.

Per the reviewer's request, we evaluated constrained decoding on the entity typing task. The F1 score achieved is 64.0, which is slightly better than the inference w/ constraints result reported in the paper. Notably, when combining constrained decoding with ACT or finetuning, the F1 score improves significantly to over 72.0, demonstrating the effectiveness of our method.

Q1. Choosing relevance as a constraint in summarization seems a bit arbitrary.

We chose relevance as a constraint because it is a key aspect of summary evaluation identified in prior work [1,2]. While coherence is also critical and length could be considered applicable, neither falls within the constraint category f(x, y) that this experiment aims to investigate.

[1] Fabbri, Alexander R., et al. "Summeval: Re-evaluating summarization evaluation." TACL 2021.

[2] Zhang, Tianyi, et al. "Benchmarking large language models for news summarization." TACL 2024.

Q2. where is the results for extraction?

We emphasize the Constraint Satisfaction Rate (CSR) in our analysis to demonstrate the transferability of constraint adherence capabilities. Event trigger extraction has traditionally been framed as an NLU task rather than NLG, and there is no universally established metric for this specific context. However, in response to the user’s request, we report the exact match (EM) accuracy below.

Dear Reviewer dUjU,

Thank you for your time and thoughtful feedback on our paper. We hope our responses have addressed your concerns and kindly request you to consider raising your score. If there are any remaining issues, please let us know, and we will be happy to provide further responses.

Thanks!

"why this constraints and not another one"

Thank you for raising this insightful question.

TL;DR:

• It is not about selecting a specific constraint for a particular task but rather identifying a representative constraint for a broad category and then choosing a suitable task as the testbed for evaluation.
• The coherent constraint falls into the f(y) type. For this type, we chose to verify a more general constraint—option lists, which are widely used to define valid solution spaces across various tasks.

Explanation

As indicated in lines 238-242, we select representative tasks for each of the three constraint categories to ensure comprehensive coverage of distinct types of constraints. It is not about selecting a specific constraint for a particular task but rather identifying a representative constraint for a broad category and then choosing a suitable task as the testbed for evaluation. Additionally, in Section 5.2, we conduct experiments to demonstrate the transferability of constraints across tasks, showcasing their adaptability and broader applicability. Overall, our work includes five groups of experiments that span a diverse spectrum of constraints, further validating the generality and robustness of our approach.

The coherent constraint falls into the f(y) type. For this type, we chose to verify a more general constraint—option lists, which are widely used to define valid solution spaces across various tasks. This approach allows us to evaluate constraints that are broadly applicable and representative of real-world scenarios.

We appreciate the reviewer's insightful feedback. We provide a detailed response below to address the concerns and questions raised by the reviewer.

W1. show that we are not just overfitting to a specific task

Our goal is to fit user-specified constraints, not tasks. The generalizability of learned constraints has been shown in Section 5. We agree with the reviewer that reframing the additional experiments in Section 5 as main experiments would better demonstrate the method's generalizability, and we will update our paper accordingly. Currently, we have five groups of experiments showing the constraint satisfaction rate for specific tasks and also the transferability of constraints, covering a diverse spectrum of constraints.

W2.1. examples on the defined constraint categorization

For f(y) constraints, an example is shown in Figure 1, where the response must be selected from an option list (Section 4.1).
For f(x,y) constraints, examples include the relevance constraint for summarization (Section 4.2) and the extractiveness constraint for information extraction (Section 5.2).
For f({x,y}) constraints, an example is the consistency constraint for temporal QA (Section 4.3), where the answer to "happens before event A" and "what happens after event A" should have no overlap (lines 377-379). We will add concrete examples in the appendix in the updated paper shortly.

W2.2. analysis on distribution of the constraint satisfaction scores

Per the reviewer's request, we present the constraint satisfaction rate distribution for entity typing and summarization. (Please check the links pointing to anonymous figures.) The observation is that ACT and finetuning exhibit similar distributions, while the original model is significantly different.

We will add this analysis in the updated paper and acknowledge ORPO for inspiring this insightful analysis.

Section 4.1: Which kind of constraints are considered when selecting a rejected response?

If the response does not satisfy one or more of the given constraints, it will be rejected.

Section 4.2: Enhanced loss function

$\mathcal{L}_{ft} = - CSR \sum_i \log P(y_i|\mathbf{x}, \mathbf{y}_{<i})$

$\mathcal{L}_{rank} = \sum_{CSR_i < CSR_j} \max (0, P(\mathbf{y}^i|x) - P(\mathbf{y}^j|x) + CSR_i - CSR_j)$

Section 4.2: How often are the constraints verified during inference?

The constraints are verified at the end, following Cao & Wang (2021).

Cao and Wang. "CLIFF: Contrastive learning for improving faithfulness and factuality in abstractive summarization." EMNLP (2021).

Section 4.3: Footnote 6 – Could you please elaborate in a more technical way on what the ‘garbage in-garbage out’ problem is?

The original model cannot generate reasonable responses without any finetuning. In this case, no informative supervision signals can be collected by comparing the responses, as none of them are of high quality.

Section 4.3: How is this conflict defined, and how is it quantified?

The conflict is defined as the overlap of events in two responses. It is quantified as the ratio of overlapping events.

Section 5.2: When the source task = ‘-’, what does this mean?

This indicates that inference was done over the base model (before ACT training).

Dear Reviewer HmdB,

As the discussion period is ending, we would like to thank you for volunteering your time to review our paper. We hope to have answered all your questions and addressed the rest of the concerns you had.

Thanks!

We appreciate the reviewer's insightful feedback. We provide a detailed response below to address the concerns and questions raised by the reviewer.

W1. The gains are small compared to fine-tuning

We want to clarify that fine-tuning is not a baseline but rather an “upper-bound” reference. While ACT does not require human-annotated data, fine-tuning relies on human annotations (of the same data size), as mentioned in lines 280-282. The comparison with fine-tuning demonstrates the informativeness of feedback from constraint verifiers and the effectiveness of constraint-driven alignment.

ACT only requires pre-defined constraint verifiers to automatically generate supervision signals. As discussed in Section 5.1, identifying constraints demands significantly less effort than manual annotation and is similar to designing annotation guidelines.

W2.1. It would be helpful if authors can provide justification for the choice of three benchmarks.

As indicated in lines 238-242, we select representative tasks for each of the three constraint categories. We also conduct additional experiments to demonstrate constraint transferability in section 5.2. Overall, we present five groups of experiments, covering a diverse spectrum of constraints.

W2.2. constrained decoding should be compared as a baseline

Constrained decoding is orthogonal to our work and is considered one type of inference with constraints in our paper. (In other words, constrained decoding is for inference, while ACT is for training.) We have already presented the results of such methods. We show that ACT and inference with constraints are complementary. By combining our method with inference with constraints, one can achieve better performance. As noted in Footnote 3, we have tested various inference w/ constraints methods and observed no significant performance difference. For consistency, we report the results of using inference w/ constraints methods derived from constraint verifiers.

Per the reviewer's request, we evaluated constrained decoding on the entity typing task. The F1 score achieved is 64.0, which is slightly better than the inference-with-constraints result reported in the paper. Notably, when combining constrained decoding with ACT or finetuning, the F1 score improves significantly to over 72.0, demonstrating the effectiveness of our method.

We will update our paper soon to include a discussion of [1] and COLD.

Q1. how many constraints exist for each task

To avoid confounding factors, we focus on at most two constraints of the same type for each experiment and report the constraint satisfaction rate for each independent constraint.

Q2. Do you have any ideas how your approach could be further improved to address such tasks ( where verifiers are difficult to build or are expensive to run)?

As discussed in Section 5.1, implementing constraint verifiers demands significantly less effort than manual data annotation and is similar to designing annotation guidelines. Constraint verifiers are reusable (even across tasks, as shown in Section 5.2) and can generate substantial supervision signals after their initial implementation. Additionally, given a new user request, one can retrieve the corresponding constraint verifiers or the trained constraint adapters for efficient alignment, as discussed in section 5.4.

We thank the reviewer for the insightful comments and suggestions, which can help us improve the quality and clarity of our paper.

Dear Reviewer mNHe,

Thank you for your time and thoughtful feedback on our paper. We hope our responses have addressed your concerns, and we kindly request that you consider updating the score accordingly. If there are any remaining issues, please let us know, and we will be happy to provide further clarification.

Thanks!

Regarding the constraint decoding baselines.

We appreciate the reviewer’s insightful feedback. The key advantage of ACT-style tuning methods over inference-time interventions lies in their ability to enhance the model’s capabilities, whereas inference-time interventions rely on utilizing the existing model capabilities. This partially explains why the two methods are complementary.

In our paper, we evaluate different inference-time intervention methods based on the literature related to each task. For entity typing, we assessed post-hoc rule-based correction and constrained decoding. For summarization, we employed reranking beams at the last decoding step, a proven method that is effective in boosting model performance (Cao and Wang, 2021). The results align: ACT can further boost model performance. Notably, ACT can improve constraint satisfaction rates, achieving performance levels comparable to inference-time interventions.

We also compared constrained decoding and ACT on a subset of the CommonGen validation set. Constrained decoding achieved a ROUGH-L score of 41.6, while ACT, after less than 300 training steps, achieved a score of 42.0, further demonstrating the effectiveness of ACT. Additionally, we observed that the constraint satisfaction rate (CSR; i.e., concept coverage in this case) for constrained decoding is highly dependent on the beam size, whereas ACT can achieve a CSR of 92.3% without requiring further intervention. This highlights the different advantages of ACT and constrained decoding.

Cao and Wang. "CLIFF: Contrastive learning for improving faithfulness and factuality in abstractive summarization." EMNLP (2021).

implementing constraint verifier can be a limitation

Thank you for highlighting this concern. We apologize for any confusion caused. We agree that constructing constraint verifiers is a limitation of ACT, and we have discussed this limitation and proposed potential solutions in Appendix D.

Constraints are prevalent in NLP tasks, and the extensive literature on these tasks serves as a valuable resource for identifying well-defined constraints [1-6] (more examples are available in lines 90-102). For creative text generation, for instance, a lexical-level creativity scorer [7] could be a potential tool for building constraint verifiers. Drawing an analogy to data annotation, we posit that specifying constraints is a prerequisite for tasks requiring them, as humans must first understand the task constraints before annotation begins.

At present, our approach relies on human efforts for constraint identification and verifier implementation. However, we envision the possibility of modularizing this process in the future. By combining different units, such as rule checkers and scorers, intelligent agents could potentially automate the creation of constraint verifiers, reducing the dependency on human intervention. This modular approach could streamline the workflow and expand the applicability of ACT to a broader range of tasks.

[1] Chang, Ming-Wei, Lev Ratinov, and Dan Roth. "Guiding semi-supervision with constraint-driven learning." ACL 2007.

[2] Wang, Haoyu, et al. "Joint constrained learning for event-event relation extraction." EMNLP (2020).

[3] Jang, Myeongjun Erik, and Thomas Lukasiewicz. "Consistency analysis of chatgpt." arXiv preprint arXiv:2303.06273 (2023).

[4] Pan, Wenbo, et al. "A preliminary evaluation of chatgpt for zero-shot dialogue understanding." arXiv preprint arXiv:2304.04256 (2023).

[5] Parikh, Ankur P., et al. "ToTTo: A controlled table-to-text generation dataset." EMNLP (2020).

[6] Porteous, Julie, and Marc Cavazza. "Controlling narrative generation with planning trajectories: the role of constraints." ICIDS 2009.

[7] Kuznetsova, Polina, Jianfu Chen, and Yejin Choi. "Understanding and quantifying creativity in lexical composition." Proceedings of the 2013 conference on empirical methods in natural language processing. 2013.

Dear Reviewer mNHe,

As the discussion period is ending, we would like to thank you for volunteering your time and engaging in discussion. We found your comments to be the most challenging and rewarding, motivating some significant changes and hopefully improvements to our paper. We hope to have answered all your questions and addressed the rest of the concerns you had.

Thanks!

We appreciate the reviewer's insightful feedback. We provide a detailed response below to address the concerns and questions raised by the reviewer.

W1.1. generalizability and reliance on the constraint verifier

As discussed in Section 5.1, implementing constraint verifiers demands significantly less effort than manual data annotation and is similar to designing annotation guidelines. Constraint verifiers are reusable (even across tasks, as shown in Section 5.2) and can generate substantial supervision signals after the initial implementation.

All constraints discussed in our paper are derived from general task guidelines, such as option lists, extractiveness, and consistency. Constraints are widely observed in NLP tasks [2-7] (more examples are available in lines 90-102), as discussed in Section 5.1. It is very unlikely that one cannot specify a constraint for a task unless the solution space is arbitrary text.

W1.2. constraint following is directly tied to the end task performance metric, which is often not the case

We want to clarify that, in terms of utility, any user-specified constraint is part of the task goal, narrowing down the solution space, regardless of whether it is directly or indirectly related to the performance metric. A response that does not meet the constraints is unhelpful to the user. From this perspective, [1] (such as apology and refusal rules) also falls into one type of constraint, f(x, y), as defined in our paper.

We want to kindly point out that [1] was released after the initial release of our paper. We will update our paper soon and add a discussion of this work. Note that [1] and our work have different focuses. We investigate common constraints in NLP tasks, categorize them into three classes, and propose a unified and efficient constraint-driven alignment framework.

[1] Rule Based Rewards for Language Model Safety Tong Mu, Alec Helyar, Johannes Heidecke, Joshua Achiam, Andrea Vallone, Ian Kivlichan, Molly Lin, Alex Beutel, John Schulman, Lilian Weng

[2] Chang, Ming-Wei, Lev Ratinov, and Dan Roth. "Guiding semi-supervision with constraint-driven learning." ACL 2007.

[3] Wang, Haoyu, et al. "Joint constrained learning for event-event relation extraction." EMNLP (2020).

[4] Jang, Myeongjun Erik, and Thomas Lukasiewicz. "Consistency analysis of chatgpt." arXiv preprint arXiv:2303.06273 (2023).

[5] Pan, Wenbo, et al. "A preliminary evaluation of chatgpt for zero-shot dialogue understanding." arXiv preprint arXiv:2304.04256 (2023).

[6] Parikh, Ankur P., et al. "ToTTo: A controlled table-to-text generation dataset." EMNLP (2020).

[7] Porteous, Julie, and Marc Cavazza. "Controlling narrative generation with planning trajectories: the role of constraints." ICIDS 2009.

W2. The proposed training approach is very similar to standard training methods

We want to clarify that our research direction is orthogonal to the mentioned training methods. The research question we address is: If human annotation is unavailable and models cannot perform well on a task, where should the supervision signals come from for customized alignment? Our insight is that the alignment process can be constraint-driven. We are the first to connect LLM alignment with constraint-driven learning. To support this insight, we formally categorize existing constraints into three types, demonstrate how to incorporate them into the alignment process, and show the effectiveness and transferability of representative constraints. Our findings can definitely be extended, and our constraint verifier can be integrated into other alignment processes.

Q1. Could you clarify the necessity of only evaluating on LLMs following the Apache 2.0 license … this is understandable if there is no further options.

We make this technical choice due to institution-wide policy restrictions.

Q2. How is Inference w/ constraints implemented?

As introduced in lines 265-277 and 344-346, inference with constraints is derived from constraint verifiers, but instead of assessing constraint satisfaction rate, we use them to improve the response. For summarization, we adopt the constraint verifier to rerank multiple sampled summaries, following Cao & Wang (2021).

Cao and Wang. "CLIFF: Contrastive learning for improving faithfulness and factuality in abstractive summarization." EMNLP (2021).

Q3. I'm particularly curious about the improvement

The improvement depends on the initial task performance and the properties of the constraints. Note that for entity typing and summarization, we use constraints from two different categories to demonstrate the generalizability of our framework. Our goal is not to achieve state-of-the-art performance but to formulate the concept of aligning with constraints and prove its effectiveness across a wide range of scenarios.

Dear Reviewer Fejv,

Thank you for your time and thoughtful feedback on our paper. We hope our responses have addressed your concerns, and we kindly request that you consider updating the score accordingly. If there are any remaining issues, please let us know, and we will be happy to provide further clarification.

Thanks!

Dear Reviewer Fejv,

As the discussion period is ending, we would like to thank you for volunteering your time to review our paper. We hope to have answered all your questions and addressed the rest of the concerns you had.

Thanks!