GenerationPrograms: Fine-grained Attribution with Executable Programs

We thank the reviewer for the helpful comments and acknowledgment of our method’s effectiveness and flexibility.

Table A. Statistics of the generated output.

W1. End-task accuracy

The outputs of the two methods differ stylistically, which impacts evaluation metrics. We present statistical analyses of their outputs using NLTK in Table A and observe that GenerationProgram tends to produce longer texts compared to ALCE. A plausible explanation is that ALCE uses answer-level in-context learning (ICL), directly guiding the model to generate outputs closely matching the expected answer format. In contrast, GenerationProgram employs program-level ICL, without a strict guarantee that executing the generated program will produce outputs matching the desired final form. To further verify and mitigate stylistic biases, we employ an LLM-based correctness metric introduced by Zhang et al. (2024a), shown to correlate highly with human judgments. Specifically, we compute average correctness scores for ALCE and GenerationProgram in the LFQA sentence-level setting, where the two methods exhibited the largest discrepancy. While Rouge-L, the standard metric, yielded scores of 39.9 for ALCE and 32.3 for GenerationProgram, the more nuanced LLM-based correctness metric produced much closer results—87.5 and 86.7, respectively. This suggests that when evaluated using a more robust and human-correlated metric, the difference between the two methods is negligible. Nonetheless, to maintain comparability with past work, we continue to report, we continue to report EM and Rouge-L in the main table, and we will include this discussion in the final camera-ready draft.

W2.Coherence and fluency

We agree with the reviewer that in our current setup, each sentence is generated independently, and we do not include an explicit contextualization step across sentences. To assess the coherence and fluency of our method despite this limitation, we utilized LLM-based evaluators on the LFQA dataset. Specifically, we employed the G-Eval prompt [1], which has been shown to correlate well with human judgments for evaluating text quality. We observed fluency scores of 1.3 (on a 1–3 scale) for both ALCE and GenerationProgram, and coherence scores of 3.0 (on a 1–5 scale) for both methods. These results suggest that, in practice, there is no significant difference in sentence-level coherence or fluency between the two approaches. We thank the reviewer for the thoughtful suggestion to incorporate a contextualization step. While having a secondary prompt to rewrite the output would break the attribution chain, we are currently implementing the approach of “feeding previously generated sentences during program execution” and hope to update with results on LFQA before the end of rebuttal period.

[1] G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, Chenguang Zhu

We thank the reviewer for the helpful comments and appreciate recognizing our method’s "clear novelty" and "large empirical win."

W1+Q1: Accuracy Tradeoff

The outputs of the two methods differ stylistically, which impacts evaluation metrics. We present statistical analyses of their outputs using NLTK in Table A and observe that GenerationProgram tends to produce longer texts compared to ALCE. A plausible explanation is that ALCE uses answer-level in-context learning (ICL), directly guiding the model to generate outputs closely matching the expected answer format. In contrast, GenerationProgram employs program-level ICL, without a strict guarantee that executing the generated program will produce outputs matching the desired final form. To further verify and mitigate stylistic biases, we employ an LLM-based correctness metric introduced by Zhang et al. (2024a), shown to correlate highly with human judgments. Specifically, we compute average correctness scores for ALCE and GenerationProgram in the LFQA sentence-level setting, where the two methods exhibited the largest discrepancy. While Rouge-L, the standard metric, yielded scores of 39.9 for ALCE and 32.3 for GenerationProgram, the more nuanced LLM-based correctness metric produced much closer results—87.5 and 86.7, respectively. This suggests that when evaluated using a more robust and human-correlated metric, the difference between the two methods is negligible. Nonetheless, to maintain comparability with past work, we continue to report, we continue to report EM and Rouge-L in the main table, and we will include this discussion in the final camera-ready draft.

Additionally, we further analyze the accuracy tradeoff resulting from refinement by computing Rouge-L scores for LFQA after applying refinement. Specifically, the Rouge-L score for ALCE decreases from 39.9 to 36.3, while for GenerationProgram, it changes slightly from 32.3 to 32.4. The observed decrease for ALCE highlights the inherent tradeoff between accuracy and attribution, a phenomenon also noted in GenerationProgram in Lines 218–220. Importantly, applying refinement to GenerationProgram outputs does not lead to any further significant accuracy reduction.

W2+Q2: Open-source models

We conducted the main experiment using the Llama 3.3 70B model, with results presented in Table A. These findings exhibit the same trends as our primary experiments conducted with GPT-4o, described in Section 4.1. Specifically, we observe that: (1) extractive summarization effectively enhances accuracy, indicating that open-source models are similarly influenced by long-context input; (2) GenerationProgram significantly improves attribution quality; and (3) combining extractive summarization with GenerationProgram balances accuracy and attribution quality effectively. We will include these findings in the final camera-ready version.

W3+Q8: Dataset scope

We selected the two datasets because they are among the most commonly used and challenging benchmarks for question-answering tasks. We will include additional datasets, such as summarization, e.g. MultiNews similar to Attribute-First (Slobodkin et. al., 2024), before the end of the rebuttal period.

W4: Evaluation dependence on LLMs and potential self-evaluation bias

We employed established automatic metrics that have shown high correlations with human judgments (Honovich et al., 2022; Zhang et al., 2024a; Lines 191–194). To further address potential concerns about self-evaluation bias, we conducted additional experiments during the rebuttal phase using the Llama 3.3 70B model. When evaluated with GPT-4o-based metrics (Table 1), our method demonstrates consistent performance gains, suggesting that the strong performance of GenerationPrograms is not due to a self-evaluation bias (since here we are evaluating Llama3 outputs with a GPT4o evaluator).

We thank the reviewer for the helpful comments and appreciate recognizing our method as "novel and practical" with "strong empirical improvements."

Running on open-source models

We conducted the main experiment using the Llama 3.3 70B model, with results presented in Table A. These findings exhibit the same trends as our primary experiments conducted with GPT-4o, described in Section 4.1. Specifically, we observe that: (1) extractive summarization effectively enhances accuracy, indicating that open-source models are similarly influenced by long-context input; (2) GenerationProgram significantly improves attribution quality; and (3) combining extractive summarization with GenerationProgram balances accuracy and attribution quality effectively. We will include these findings in the final camera-ready version.

ALCE showing attribution F1 exceeding 80

We note that our setting employs 10 passages (Line 178). In the ALCE paper, the best-performing model used only 5 passages. Specifically, the original ALCE results for the Summ (10-psg) setting—which closely aligns with our ALCE + extractive summarization configuration—show attribution precision and recall of 68.9 and 61.8, respectively. These figures are consistent with our reported citation F1 score of 66.4 in Table 1.

The modules are primarily summarization-focused, limiting applicability to tasks involving more complex reasoning or knowledge manipulation

We agree that our modules, though initially inspired by summarization tasks (Line 133), are fundamentally designed for general text manipulation tasks. Our evaluation explicitly includes challenging long-form question-answering tasks (Section 3.1), demonstrating their applicability beyond summarization. Moreover, as indicated in our future work discussion (Lines 347–355), we plan to extend our module set to include additional text manipulation functionalities, such as decontextualization and text simplification. We acknowledge that tasks requiring complex reasoning, multi-hop inference, or logical puzzle-solving extend beyond pure text manipulation and are indeed outside the scope of our current framework. We appreciate this important distinction and will clarify the intended scope explicitly in our paper.

Comparison against other generation pipelines

We would like to clearly discuss the differences between GenerationProgram and other works. The most critical distinction lies in our use of optional natural language instructions within each program step (Lines 146–157), allowing the planner to communicate intent to individual modules. This design enables greater flexibility, generality, and interpretability in reasoning. We illustrate its benefits in Figure 4: although argument-based programs (e.g., sentence IDs) may indicate relevance, they often obscure the precise reasoning—especially when the same source (e.g., S3) supports multiple sentences in different ways. For instance, the first sentence may draw on the “starting points,” while the second depends on length and endpoints. By incorporating explicit instructions, users can more easily verify and understand how content is generated.

In comparison:

• Attribute-First (Slobodkin et al., 2024) is limited to a single fusion operation between two sentences (Line 327). In contrast, our framework supports a more general set of operations, including paraphrasing, extraction, and compression, enabling richer composition and modularity.

• Summarization Program (Saha et al., 2023) does employ multiple modules, but requires fine-tuning on synthetic program-labeled data—a process involving complex preprocessing to identify valid programs (Lines 329–335). In contrast, our method is training-free, relying solely on in-context learning (Lines 121–126). This aligns with the reviewer’s observation about the benefits of using LLMs “without the need for retraining,” making GenerationProgram a lightweight and easily deployable solution. Moreover, while Summarization Program focuses solely on summarization tasks, we demonstrate that GenerationProgram generalizes well to long-form question answering, highlighting its broader applicability.

We will incorporate this comparative discussion in the final camera-ready version to more clearly delineate our contributions.