SciPIP: An LLM-based Scientific Paper Idea Proposer

Overall, this is an interesting study. However, I still have some concerns:

Large language models may rely on knowledge from their training data when generating ideas. If the model has "seen" content similar to existing research, it could end up producing ideas that are essentially variations of what’s already out there. So, a high similarity score might just mean the model is "copying" known information, rather than truly generating something new.

In many cases, the model’s similarity scores don’t give enough insight to tell if an idea is truly innovative. A high similarity score might just mean the idea makes minor tweaks to existing work, while a low score could indicate it’s different—but if it doesn’t align with the actual research question, that difference isn’t useful. This makes the use of similarity scores quite limited, and it calls for other tools to fill in the gaps.

Since similarity scores are based on semantic overlap, they don’t really measure the quality of the idea itself. Semantic similarity only captures how similar the wording is, not the underlying logic, originality, or technical feasibility. Relying on these scores alone could lead to conclusions that miss the actual value of the ideas.

One potential concern is the reliability of using large language models (LLMs) to evaluate the originality of ideas generated by SciPIP. Since LLMs may have been trained on a vast corpus that includes the same or similar literature, there could be a risk of generating ideas that are not genuinely novel. This raises questions about the ability of LLMs to accurately assess the originality and publishability of these generated ideas. Given this limitation, would it be more effective to supplement the evaluation with alternative methods, such as expert reviews? These approaches could help to ensure that SciPIP’s outputs demonstrate true innovation and relevance to the research community.

The current evaluation mainly relies on comparing generated ideas to existing papers using similarity scores, but this approach has some limitations in assessing true novelty and academic value. A high similarity score doesn’t necessarily mean the idea lacks innovation, and a low score doesn’t always indicate a breakthrough. Since the similarity score is based on semantic matching, it may miss aspects like technical feasibility or practical relevance, making it hard to capture the full academic contribution. Additionally, without user studies or human expert evaluation, it’s challenging to verify the system’s real-world applicability, which may weaken its persuasive power. This could make publishing as a research paper more challenging, especially for venues that prioritize metrics of academic innovation. Positioning it as a demo paper or incorporating more user studies could better highlight the system’s practical usefulness and potential impact.

it would be better to cite and discuss the following paper: Li, Weitao, et al. "Citation-Enhanced Generation for LLM-based Chatbot." arXiv preprint arXiv:2402.16063 (2024).

Would this paper be more suitable as a demo paper? Alternatively, would it be better to conduct additional user studies and shift toward an HCI-focused direction? Given the current evaluation approach, it seems that positioning it as a research paper may be more challenging.

1. In SciMON, a related paper on scientific idea discovery, semantic, entity, and citation signals are also leveraged during literature retrieval. Although SciPIP extends SciMON's framework with additional techniques and demonstrates improved performance, the enhancements are generally minor and come with extra computational costs and complexity.

2. While the three pipelines for idea generation are interesting, they lack a clear explanation of the underlying intuition for each submodule. An ablation study is also missing, leaving unclear how each submodule contributes to the overall performance.

3. The evaluation pipeline raises further concerns. For instance, in assessing similarity with ACL papers, it’s unclear how well an LLM can judge similarity between ideas, given that ideas can align across different dimensions—such as background, settings, or methods. Simply prompting the LLM to provide a similarity score appears insufficient. Additionally, the paper outlines several modules necessary to prompt LLMs to generate reliable research ideas. Thus, it seems unlikely that LLMs alone can accurately determine novelty, suggesting that scoring generated ideas solely based on a novelty score from an LLM may not be reliable.

4. The reviewer also questions the quality of generated ideas. For example, in Figure 4, SciPIP generates an idea for multimodal learning that involves “minimizing a contrastive loss function to bring together similar instances of different modalities,” a widely accepted approach in the field (e.g., CLIP [1], published in 2021). Furthermore, the matched ground-truth idea discusses data augmentation, a concept omitted in the generated idea, yet the LLM assigns a similarity score of 4.

[1] Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., ... & Sutskever, I. (2021, July). Learning transferable visual models from natural language supervision. In International conference on machine learning (pp. 8748-8763). PMLR.

Many aspects of the evaluation are delegated to GPT-4o. It is not clear how, for instance, "novelty" and "feasibility" are determined except that GPT-4o is used for the evaluation.

Also, SentenceBERT may not be the best option to evaluate the similarity between entities and textual fragments, especially if the pre-computed models are used. It is known that SentenceBERT has some limitations when comparing nuanced semantics.

The paper's approach to aid researchers through enhanced literature retrieval and idea suggestion is noteworthy but not path-breaking. It represents a solid application of existing techniques like LLMs and embedding-based retrieval methods but lacks a transformative innovation to elevate its contribution to the research community.

Overall, while useful, SciPIP as presented currently may be best suited for practical use in specific contexts rather than serving as a major research contribution.

Some parts of the paper are not detailed well enough and make it difficult to follow the understanding of the paper (see questions).

1. I am very concerned about the insufficient related work section. It is very short and does not cover many important aspects. For example, there are many other existing scientific literature database, arXiv, pmc, pubmed, semantic scholar, with much larger scale (over millions of articles), but they are not cited. There have been many advancements in the field of neural information retrieval with state-of-the-art embedding models such as OpenAI-v3, Grit-LM, Rank-Llama, but the authors choose to use a fairly out-of-date embedding model (sentence-bert) with a much shorter context window. There are also many works on using LLM directly to retrieve, search and select scientific literature such as RankGPT, this method is compared with other GPT-based retrieval strategies in BIRCO, a benchmark that includes a scientific literature benchmark, DORIS-MAE. None of these works are mentioned. At least some of them should be used as baseline models to compare with the paper's proposed retrieval strategy.

2. The second concern is the scale and comprehensiveness of this work. This work only focuses on a very narrow field in computer science, namely AI/ML, and its experiment setting is on an even narrower field, NLP. I would be fine with these settings except for the fact that the authors named their paper "scientific paper idea proposer". The word scientific implies a much larger domain, including biomedicine, physics, math, computer science and other quantitative sciences. Because the scale of this work is severely limited, I am doubtful for its contribution to the research community. FYI, these limitations about scales and lack of comprehensiveness should be openly acknowledged in the limitation section (which is also insufficient and very short).

3. The third concern is the lack of originality of this work. It is true that the problem this paper is trying to tackle is very interesting and challenging (i.e. scientific idea creation). However, this paper does not use sufficiently innovative approaches to solve it. The entire retrieval strategy is an ensemble of previously utilized techniques, such as entity matching, semantic embedding search, co-citation (this is implemented in many existing databases such as PubMed), and clustering. The idea proposal pipeline is only an implementation but with limited originality. In my opinion the most original aspect of this paper is its introduction of the annotated database where each paper has its ideas extracted and processed, though the paper does not talk about quantitative quality control of the database creation process.

4. I have some questions/doubts about the evaluation section in the paper, I will reserve them to the question section.