GraphEval: A Lightweight Graph-Based LLM Framework for Idea Evaluation

Q1. While the approach works well on specific datasets (ICLR Papers and AI Researcher), the generalizability to other domains or different types of research content is unclear. Given the topic of the paper is research idea validation, the experiment turned out to detect whether this paper will be accepted. The claim and experiment are a bit stretched.

Response: Thanks for the reviewer’s constructive feedback. We believe that research idea validation is an important and urgent topic to study. Existing tasks or datasets for idea evaluation often rely on employing human evaluators to assign labels to ideas, a method that is not only costly but also makes it difficult to acquire large training datasets. Therefore, we focus on tasks and datasets from open reviews, such as those at ICLR. Given that ICLR reviewers are generally of high caliber, this dataset not only yields high-quality labels but also provides a substantial amount of free data. The research tasks associated with these datasets are also of widespread academic interest. Additionally, we focus on whether papers are accepted for two reasons: 1. This data is relatively easy to obtain and abundantly available; 2. Other tasks, such as predicting the average score of papers, can be subjective due to varying scoring standards among reviewers, whereas the acceptance or rejection of papers tends to be more objective.

On the other hand, inspired by your feedback and Reviewer zpR7's Q1, we attempt to validate our method in a long-form text evaluation task to test its generalizability to other domains or different types of research content. Specifically, we used human-annotated data from the FActScore dataset [1], where each entry contains "atomic facts" about celebrities generated by LLMs, along with assessments from human annotators on whether these "atomic facts" were supported by the materials provided to the annotators. Based on the "atomic facts" and human annotations from the training set, our method needed to predict the labels of "atomic facts" in the test set that were partitioned off. We selected topics such as Ramesses IV, Lanny Flaherty, and Florencia Bertotti, and divided the training, validation, and test sets in a 7:1:2 ratio. We compared GraphEval and some applicable baselines on this dataset in the following table. The experimental results in the table verify that our approach performs well on the long-form text evaluation task, demonstrating good adaptability to various tasks. All the discussions and experimental results mentioned above have been updated in Section E.1 of the Appendix and Table 20 of the revised PDF version.

Comparative performance results on the Fact Verification dataset. Bold text denotes the best results.

[1] FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation, EMNLP 2023.

Q2. While GraphEval-GNN offers significant improvements in performance, its scalability is questionable for larger datasets. Training GNNs on extensive viewpoint-subgraphs may become computationally prohibitive, especially if the graph structure grows significantly.

Response: Thanks for your valuable feedback. Indeed, in some standard and mature GNN tasks such as OGB [1,2], the number of nodes and edges can reach the order of hundreds of millions. In real-world large-scale deployment scenarios like recommendation systems, GNNs can efficiently handle tens of billions of nodes and trillions of edges [3,4]. The number of nodes in our viewpoint-subgraphs often does not reach the million level, so technically, training GNNs on extensive viewpoint-subgraphs is not problematic.

Furthermore, our paper also provides another method that performs slightly worse than GraphEval-GNN but is more efficient and computationally friendly, called GraphEval-LP. This method is based on label propagation and has a linear time complexity of $O(k \cdot n \cdot p) \ll O(n^2)$, where $k$ is the number of iterations, $p$ is the predefined maximum degree for each node, and $n$ is the number of nodes. Therefore, as the number of viewpoint-subgraphs increases, GraphEval-LP can efficiently perform idea evaluation.

[1] Open graph benchmark: Datasets for machine learning on graphs, Neurips 2020.

[2] Graph neural networks: A review of methods and applications, AI Open 2020.

[3] Graph convolutional neural networks for web-scale recommender systems, KDD, 2018.

[4] Pregel: a system for large-scale graph processing, KDD 2010.

Q1. The paper lacks a thorough theoretical analysis or justification for the proposed mechanisms, such as the viewpoint-graphs extraction and the plagiarism detection mechanism. Whether viewpoints extracted by LLMs are reasonable? If so, how to verify the accuracy of these viewpoints? Is there an overlap between viewpoints from different complex ideas? How are the inherent LLM issues—like hallucinations and token limitations—addressed or mitigated in this framework?

Response: Thanks for the reviewer’s insightful questions. We answer the questions step by step.

[Theoretical analysis or justification] Our paper primarily focuses on an empirical analysis of how graph modeling can enhance the performance of LLMs in idea evaluation. Due to the difficulty in constructing rigorous theoretical proofs and analyses in most areas of LLM research, our paper primarily focuses on conducting extensive experimental analysis to substantiate our conclusions.

[Verification of the accuracy of these viewpoints] Verifying the accuracy of these viewpoints is an open question in academia, often requiring well-grounded labeled data or time-consuming and labor-intensive human evaluation. However, this is not the main focus of our paper; our core interest lies in the evaluation of idea quality.

[Overlap between viewpoints] It is possible for viewpoints from different complex ideas to overlap. In our paper, this aspect is modeled using similarity edges between viewpoint nodes, where edges between overlapping viewpoint-nodes have similarity weights closer to 1.

[Solution for inherent LLM issues] In the introduction, specifically in Figures 1 and 2, we compare the performance of LLMs and GraphEval. While current LLMs may introduce hallucinations and biases when processing complex and subjective ideas, GraphEval deconstructs these ideas into simpler, more comprehensible viewpoints. Using a graph-based framework, it provides more objective reasoning, effectively mitigating illusion issues in idea evaluation. As for token limitations, we believe this is mainly the focus of papers on long-context LLMs, not the central concern of our paper.

Q2. The paper seems to underutilize the reasoning and generative capabilities of LLMs, which could improve GraphEval’s interpretability and effectiveness in idea evaluation tasks.

Response: Thanks for the reviewer’s constructive feedback. Indeed, we have explored the reasoning and generative capabilities of LLMs like COT/TOT in our baselines. However, the results indicate that these capabilities currently have little impact on LLMs for idea evaluation. We believe that in the future, LLMs with larger parameters will possess more powerful reasoning and generative capabilities to effectively address the idea evaluation problem. However, these larger LLMs, compared to smaller ones, will incur higher inference costs in idea evaluation. Therefore, our framework essentially aims to utilize lightweight graphs to assist smaller LLMs in viewpoint-level reasoning, thereby enabling smaller models to achieve good results in idea evaluation.

Q3. Several typos are present in the paper. For example, In line 302, "... As illustrated in Sec 3, ..." lacks a period after "Sec". In lines 414-422, the use of macro F1 score to evaluate the accuracy is inconsistent with the content "..., and micro F1 score ...." (in lines 399-401).

Response: We apologize for the confusion. We have revised them in the current version: "micro" is corrected to "macro" and "Sec" is corrected to "Sec.".

Q4. Could the authors discuss the similarities and differences between GraphEval and GraphRAG [1]? GraphRAG breaks a document into chunks, extracts a knowledge graph from raw text, builds a community hierarchy, generates summaries for these communities, and leverages these structures in RAG tasks. This seems similar to the viewpoint-graph extraction proposed here.

Response: Thank you for your constructive questions. Indeed, the two methods seem similar, both aiming to better manage text in the form of graphs. However, they clearly differ in many aspects:

[Element]: The constituent elements of GraphEval are viewpoints, whereas the constituent elements of GraphRAG are raw text chunks.

[Construction of the graph]: GraphEval uses text similarity and graph algorithms, while GraphRAG employs LLMs to identify relationships, which is costly and slow.

[Applications]: GraphEval is used to evaluate ideas and some hard-to-understand texts, while GraphRAG is designed for long-context QA.

[Technique]: GraphEval utilizes a graph algorithm for output, whereas GraphRAG primarily uses a rag approach.

The authors’ rebuttal addressed most of my concerns, so I have raised my score from 5 to 6.

Thanks for the authors' comprehensive evaluation of viewpoint-graphs extraction from two perspectives, which effectively addressed my concerns. Additionally, could authors release your related code, which ensures the reproducibility of this paper?

I am happy to see the paper accepted (raising my score from 6 to 8).

Thank you for the reviewer's insightful feedback and hard work. Your valuable suggestions have significantly improved our paper. We are pleased to hear that our responses have addressed most of your concerns. During the rebuttal phase, we followed the reviewers' suggestions and conducted numerous new settings and experimental trials. Consequently, we are continuously organizing the code and adding necessary comments to ensure its readability and usability. We commit to including the code link in the potential camera-ready version of the paper to guarantee its reproducibility. We are committed to incorporating the suggested changes in our revisions to further enhance the manuscript. By the way, as Thanksgiving is approaching, we also sincerely wish you a Happy Thanksgiving. Thank you for the valuable feedback and hard work you provided during the rebuttal phase.

Q1. In the domain of automated paper review generation, several existing works relevant to this field are missing from the discussion [1-3].

Response: Thank you for your feedback. In the introduction and related work sections of our paper, we have cited and discussed a vast array of representative works on idea/text evaluation. As a critically important field, idea evaluation is bound to attract widespread attention in academia, and inevitably there will be some works that we cannot discuss and cover. Our paper currently focuses primarily on several LLM-based works, which represent the latest research trends in academia.

However, following the reviewer's suggestion, we have added a discussion of these three automated paper review generation works in the related work section: "In addition, numerous studies employed fine-tuned lightweight language models (e.g., BERT (Devlin, 2018)) to evaluate complex texts, such as dialogues (Thida, 2021), tweets (Pota et al., 2021), and the novelty of ideas (Just et al., 2024; Yuan et al., 2022). Conversely, recent studies have sought to leverage the domain knowledge and logical capabilities of LLMs to create idea evaluators(Ubonsiri, 2024; Baek et al., 2024; Du et al., 2024; Lin et al., 2023a). Du et al. (2024) proposed using a prompt-based approach to allow LLMs to act as reviewers and meta-reviewers in order to assess the level of papers/ideas based on different evaluation criteria.".

Q2. The paper contains informal language in some variable names, such as "max_iters" and "cos sim".

Response: Thanks for your comments. In fact, these concepts are quite basic and are often used in graph-related applications. We use this language because it facilitates the understanding of the readers. However, following the reviewer's suggestion, we have made them more formal in the revised version: "Then, we compute the cosine similarity $s$ between their embeddings: \{s}(e_i, e_j) = \frac{e_i \cdot e_j}{\|e_i\| \|e_j\|}\. We perform multiple iterations of label propagation on graph $G$ until the labels no longer change."

Q3. The framework represents edges between viewpoints within a single document and across multiple documents as undirected. Could the authors clarify how (or if) temporal edges are integrated into the graph?

Response: Thanks for your insightful question. As discussed in [lines 212-232] of Section 3 of the paper, due to the sparsity of the edges constructed by the prompt-based LLM, our edges are ultimately built based on the similarity between embeddings of the viewpoint-nodes, which are undirected. We are fully aware that idea evaluation is temporal, hence in [lines 321-344] of Section 5, we introduced how we incorporate temporal information into viewpoint-nodes. Compared to integrating temporal edges into the graph, adding temporal information to the nodes’ embeddings is simpler and more practical.

Q4. In line 288, what is n when predicting the label y^hat?

Response: Thanks for your question. In [lines 200-203], we introduced that $n$ represents the number of viewpoints extracted from a given research idea. However, the distance from where y^hat​ is introduced is indeed too far. We are sorry for the confusion caused.

In the revised version, we have added further explanation: The predicted label $\hat{y}$ is then determined by selecting the dimension with the highest value in the summed vector, i.e., $\hat{y} = \arg\max_j \left( \sum_{i=1}^{k} d_i \right)_j$, where $j$ indexes the dimensions of the vector and $k$ means the number of viewpoints extracted from a given research idea.

Q5. Lines 270-274 describe the initialization process is ambiguous. The initialization node features are formed with node labels? Could the authors clarify this initialization procedure?

Response: Thanks for the reviewer’s question. We believe the reviewer has a misunderstanding about the use of label propagation in GraphEval-LP, which is a commonly used method that is simple to deploy [[1](https://en.wikipedia.org/wiki/Label_propagation_algorithm#:~:text=Label%20propagation%20is%20a%20semi,have%20labels%20(or%20classifications)].

In fact, in the initialization procedure section, we do not mention node features, and the label propagation process does not involve node features either. The process of label propagation involves transferring labels from labeled viewpoint-nodes (label initialization of the training set) through the graph to unlabeled viewpoint-nodes (testing set) [lines 270-278], and aggregating the viewpoint-nodes contained in an idea [lines 279-283] to obtain the final prediction result for the idea.

Q1. The authors mainly evaluate the proposed method on only the research idea evaluation task, and, while this task is very important and less explored, this point may limit its generalizability to other tasks and domains (i.e., I believe it can be well applicable to other tasks related to evaluating long text and it is worth trying it).

Response: Thanks for the reviewer’s constructive feedback. We appreciate the reviewer's suggestions, which are excellent for exploring the generalization capability of our method across various tasks and domains.

Therefore, following the reviewer's advice, we have chosen to conduct experiments on the dataset described in [1]. Specifically, we used human-annotated data from the FActScore dataset, where each entry contains "atomic facts" about celebrities generated by LLMs, along with assessments from human annotators on whether these "atomic facts" were supported by the materials provided to the annotators. Based on the "atomic facts" and human annotations from the training set, our method needed to predict the labels of "atomic facts" in the test set. We selected topics such as Ramesses IV, Lanny Flaherty, and Florencia Bertotti, and divided the training, validation, and test sets in a 7:1:2 ratio. We compared GraphEval and some applicable baselines on this dataset in the following table. The experimental results in the table verify that our approach performs well on the long-form text evaluation task, demonstrating good adaptability to various tasks. All the discussions and experimental results mentioned above have been updated in [Section E.1 of the Appendix] and [Table 20] of the revised PDF version.

Comparative performance results on the Fact Verification dataset. Bold text denotes the best results. For all metrics—Accuracy, Macro Precision, Macro Recall, and Macro F1 Score—higher values indicate more precise predictions.

Q2. The authors can discuss some works on evaluating the long-form text [1, 2], as some of them tend to divide the long text into multiple subsets (similar to the proposed approach) and evaluate each of them.

Response: Thanks for the reviewer’s insightful feedback. Following the reviewer’s suggestion, we add the corresponding citations and discussion in [Section 2]:“Recently, some research works have been evaluating long-form texts, such as biographies of people (Min et al., 2023) and complex mathematical reasoning texts (Lightman et al., 2023). These studies divide the long text into multiple subsets and evaluate each of them. Inspired by these works, we decompose the obscure ideas into simple, understandable viewpoint nodes using LLMs, and further evaluate the idea based on graph algorithms.”.

Q3. The authors can incorporate more analyses, to showcase the efficacy of the proposed approach more. One experiment that is worthwhile to try is, to showcase the generalizability of the proposed approach (or the constructed viewpoint graph) to new papers in 2023 by training it with papers before 2022.

Response: Thank you for your insightful suggestions. We believe that this suggestion, like Q1, can effectively help validate the generalization ability of our method across different dimensions. Following the reviewer's advice, we selected papers from before 2022 in the ICLR Papers dataset as the training and validation sets, and papers from 2023 as the test set. We compared the performance of GraphEval with other classic baselines in the following table. The results in the table validate GraphEval's temporal generalization ability in the task of idea evaluation. All the discussions and experimental results have been updated in [Section E.2] of the Appendix and [Table 21] of the revised PDF version.

Comparative performance results under the setting of idea evaluation of different years. Bold text denotes the best results. For all metrics—Accuracy, Macro Precision, Macro Recall, and Macro F1 Score—higher values indicate more precise predictions.