Response to reviewer 7Ub3

Q1. Design of unchanged candidates in the LLM decider.

Response: Thank you for carefully reading through the appendix. We apologize for not clearly explaining the the LLM decider's working process. Here, we explain the detailed designs.

With the retrieval size of $r_q$ and $t^1_q$ core citations in the candidate set, the decider needs to select $t^1_q$ candidates most likely to be core citations from the $r_q$ retrieved candidates. One approach is to directly reranking the $r_q$ retrieved candidates and select the top $t^1_q$. However, there are two shortcomings: (1) reranking all the $r_q$ retrieved candidates leads to long input context for the LLM decider, harming the performance; (2) directly reranking the $r_q$ retrieved candidates does not utilize any information from the embedding model. In other words, the $r_q$ retrieved candidates have different inner products by the embedding model, which are informative as the prediction of their core citation likelihood in the retrieval module.

We utilized the likelihood prediction in the retrieval module, regarding the $(2t^1_q-r_q)$ retrieved candidates with largest inner products as safe core citations and exempting them from reranking. Then we reranked the remaining $2(r_q-t^1_q)$ retrieved candidates with the LLM and selected the top $(r_q-t^1_q)$ ones, resulting in $t^1_q$ selected candidates in total.

For example, we used $t^1_q=5$ and $r_q=8$ in our main results. Here, the number of fixed top candidates is 2, and we reranked the remaining 6 retrieved candidates and selected the top 3. Also, we kept $t^1_q=5$ and tested $r_q=10$ in Section 4.5.2. Here, the number of fixed top candidates is 0, and we reranked all 10 retrieved candidates with the LLM agent and selected the top 5.

We will add these details in the main text of our paper. To avoid confusion, we will also modify Figure 3 accordingly, highlighting the unchanged top candidates and the reranking of the remaining ones.

Q2. Ablation study of the curriculum stage 1.

Response: Thanks for pointing out this. It is true that the performance without Stage 1 is very close to the full curriculum, especially in social science. To examine this, we use t-test to compare the full design with the ablation versions. Please refer to "result.pdf" in the global response for detailed results.

The results give us a deeper understanding of the role of each part in our designs. Generally, all parts of our designs are valid with significance ($p<0.01$ or $p<0.1$ in overall performance through t-test). Moreover, when focusing on social science papers, which only comprise a small proportion of all papers, Stage 1 of curriculum finetuning is only slightly beneficial. Therefore, when only applying to social science papers, it is an alternative for users to skip Stage 1 if they want to save computational cost with the cost of a slight performance drop. In contrast, when applying to natural science papers, it is necessary to keep Stage 1 for better performance.

Q3. Comparison to previous binary citation prediction methods.

Response: Thanks for this question. Actually, the baseline embedding models we compared are binary citation prediction methods. In the retrieval stage, either the baselines or our models encode the query and candidates into vectors and then calculate each candidate's score via inner products. The only difference is that the baseline methods have not been trained to distinguish core/superficial citations. Therefore, they only try to ensure the score as cited papers>non-citations (binary citation prediction). In contrast, our model learned to distinguish citations and non-citations in the curriculum Stage 1 and further learned to distinguish core/superficial citations in the curriculum Stage 2. Therefore, our model can try to ensure the score as core citations>superficial citations>non-citations.

Also, the ablation study of curriculum Stage 2 is the case that our model only learned binary citation prediction. The significant performance drop indicates the limitation of merely using binary citation prediction model in our core citation prediction task.

We will add the explanations to the revised version of our paper. Hopefully, our interpretations can help readers understand the relationships and differences between our model and the binary citation prediction methods.

Q4. Details of the numbers of $t_q^1$ and $t_q^2$.

Response: Thanks for this question. In this paper, the values of $t_q^1$ and $t_q^2$ are manually set to be 5. We used this setting to align with SciDocs, which is a widely applied benchmark dataset in scientific text embedding and citation prediction [1]. In SciDocs, each query is provided with a candidate set of 5 cited papers. Therefore, we extended it into a candidate set of 5 core citations ($t_q^1$) and 5 superficial citations ($t_q^2$).

Our model can directly adapt to testing sets with different values of $t_q^1$ and $t_q^2$. In all parts of our designs, our model did not fit to specific values of $t_q^1$ and $t_q^2$ but only learned to rank the candidates in the order of core citation>superficial citations>non-citations. Therefore, its performance would not be affected by the values of $t_q^1$ and $t_q^2$.

Besides, according to our statistics over 12M papers across 19 scientific fields in the Microsoft Academic Graph (MAG), on average, each paper has 10.207 ($\sigma=$0.007) core citations and 10.727 ($\sigma=$0.007) superficial citations. Therefore, it is reasonable to set $t_q^1=t_q^2$.

We will add the explanations and statistics to the revised version of our paper. Hopefully, our interpretations will make it easier for readers to understand our experimental settings.

[1] SPECTER: Document-level Representation Learning using Citation-informed Transformers. ACL 2020.

Response to reviewer b1Uz

Q1. Novelty of the core citation idea.

Response: Thanks for the literature. We agree that citation classification is not new. Existing works classified citations with traditional ML [2,3,5] and DL [1,4] according to the roles in the context (background, method, etc.) [1,2] and the author's attitude towards the citations (supporting, contrasting, etc.) [3-5]. However, they all focused on the semantics of each individual paper, which requires manual annotation, limiting the data size to thousands and consuming up to years of manual work [4].

In contrast, we defined core citations on the citation network among vast papers. For a query paper $q$ and its citations $S_q$, if there exists a subsequent paper that cites $q$ and one of $q$'s citations $s_q\in S_q$ at the same time, then we regard $s_q$ as a core citation of $q$ (Section 2.1). Instead of subjective semantic labels by manual work, our design objectively reflects the collective behavior of the scientific community, i.e., the subsequent citations of $q$ indicate the degree of recognition of $q$ and its citations $S_q$ by the community.

Thus, we classified citations from a novel perspective. Also, because our classification criteria are objective, we require no subjective manual annotations and can label large datasets by automatic structural analysis on the citation network. After obtaining the labels from the citation network, we trained our model to predict them purely based on the texts. Hopefully, we have clarified the novelty of our core citation idea compared to existing works.

Q2. Annotation of core and superficial citations.

Response: Yes, we labeled citations in the entire MAG automatically and required no manual annotations (See Q1).

Q3. Scientific contributions and compliance with NeurIPS's scope.

Response: Thanks for recognizing our contribution to the application. Here, we want to clarify that our work is not only applications of LLMs and models but includes novel scientific contributions:

• Definition and identification of core citation. AS we mentioned in Q1, our definition of core citation classifies citations from a novel aspect, which exhibits statistical rationality. Instead of massive human work, we can label large-scale datasets by automatic network structure analysis based on our idea.
• Hybrid workflow combining small and large language models. More importantly than the finetuning and prompting details, we proposed the hybrid framework incorporating small and large LMs. Small models are computationally efficient but lack reasoning capability, only capturing textual similarities, while large models have text reasoning ability but are computationally expensive and limited in context length. Our framework combines the advantages of both models, ensuring computational efficiency in large-scale tasks and maintaining the reasoning ability beyond simple textual similarities.

Besides, as mentioned in the NeurIPS call for papers, the conference's topics include:

• Applications (e.g., language, etc.)
• Machine learning for sciences (e.g., social sciences, etc.)

In this paper, we first proposed a novel and meaningful NLP task in social sciences, and then designed an applicable ML workflow that reached good performance on the proposed task, which we believe is within the scope of NeurIPS.

Also, we notice that a number of papers with similar scopes and contributions have been published in NeurIPS. For example:

• [6] designed a communicative LLM agent framework via prompting, which can generate conversational data for studying.
• [7] designed two failure modes to guide the jailbreak of LLMs, which enlightened the safe training of LLMs.

Considering these, we believe that our paper is suitable for NeurIPS, and hopefully, our interpretations can better illustrate the novelty and value of our work.

Q4. Details about manual review and revision.

Response: Thanks for the question. We only manually reviewed and revised one query paper (the selected Transformer-XL paper) and its several candidates. We used the manually reviewed and revised texts as the one-shot example for the analyzer and decider in processing other papers in the dataset, where no more manual work is needed. Please refer to Appendix A.6 for the full texts we manually reviewed and revised.

Q5. Statistical significance.

Response: Thanks for the question. We used the two-tailed t-test for statistical significance of the performance differences between different models. In detail, we tested in:

• Overall performance. Ours VS the strongest baseline: $p<0.001$ averaging all fields.
• Ablation studies. The full design VS different ablation versions: $p<0.01$ or $p<0.1$ averaging all fields.
• Other analyses. One-shot VS few-shot, GPT models VS other models. Here, we only tested 10% of the testing set to reduce computational consumption, but small enough $p$ indicates that 10% samples can reflect the performance differences without loss of statistical significance.

Q6. Minor typos.

Response: Thanks. We will check and correct these.

[1] Structural scaffolds for citation intent classification in scientific publications. NAALC 2019.
[2] Measuring the evolution of a scientific field through citation frames. Transactions of the Association for Computational Linguistics 2018.
[3] Citation context classification of a citation classic concerning citation context classification.Social Studies of Science 1988.
[4] Scite: A smart citation index that displays the context of citations and classifies their intent using deep learning. Quantitative Science Studies 2021.
[5] Automatic classification of citation function. EMNLP 2006.
[6] CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society. NeurIPS 2023.
[7] Jailbroken: How Does LLM Safety Training Fail? NeurIPS 2023.

Response to reviewer 3KQH

Q1. Overly dense terminology in Section 2.1.

Response: Thanks for pointing out this. In this paper, we follow previous computational social science [1,2] studies and define the core citations of a query paper from the citation network. We apologize for making the formulas too complicated when trying to express the intuitive definition mathematically. Here, we rephrase the terminology:

For a query paper $q$ and its citations $S_q$, if there exists a subsequent paper that cites $q$ and one of $q$'s citations $s_q\in S_q$ at the same time, then $s_q$ is among the core citations of $q$. Therefore, $q$'s core citations can be expressed as:

Hopefully, this simplified version will be easier for readers to understand.

Q2. Necessity and impact of text-based (rather than network science-based) core citation prediction.

Response: Thank you for this question. You are right that network science is helpful for identifying core citations of existing papers. As mentioned in Q1, we did utilize knowledge in network science and science of science to define the core citations given query paper $q$ based on subsequent papers that cite $q$ in the citation network. Then, we classify citations in the Microsoft Academic Graph (MAG) as core/superficial. However, this paper did not focus on analyzing these classifications of existing citations. Instead, we treat the network-based classifications as labels and train a text-based model, which can predict the core citations of $q$ without knowing subsequent paper that cites $q$.

Here are two representative scenarios where the subsequent paper that cites $q$ is unknown for the given query $q$, illustrating the meaningful applications of our text-based model:

• Paper recommendation (for ongoing papers). Our model can recommend suitable citations to researchers who are writing papers. You are right that researchers should know what the core citations in their field are. However, the rapid development of modern science brings about exponential growth in publications, where researchers may find an overwhelming number of publications that match their interests but are largely irrelevant to their needs in writing the specific paper [4]. The so-called "burden of knowledge" [2,3] makes manual identification of papers to read or cite challenging, relying on a researcher’s ability. Therefore, a lot of works have been done on paper recommendation systems [4,5]. Our model can serve as such a system, where users input their draft of the abstract as the query and set a group of possibly related papers as the candidates. Then, our model can help identify papers that are most suitable to cite in the ongoing paper. In this scenario, it is impossible to identify the core citations of the ongoing paper based on subsequent papers citing it.
• Science of science study (for existing papers). As you mentioned, there exists abundant science of science researches that utilize citation networks to analyze published papers [1,2]. However, recent studies have recognized that the subsequent citations of a paper require three to five years after publication to accumulate, and a proportion of papers, named "sleeping beauties", may experience a burst of attention after up to ten years [6-8]. Several famous studies, e.g., measuring disruption, have reported the impact of delayed citation on their analyses [1]. Therefore, although identifying core citations based merely on the citation network is feasible for old papers that have accumulated sufficient subsequent citations, it is not applicable for the latest papers that have not yet been cited. In contrast, our text-based method can identify the core citations of a newly published paper without relying on subsequent papers citing it. This allows us to analyze the latest science using the idea of core citation, which is meaningful for understanding cutting-edge trends in scientific research.

Hopefully, these interpretations can better illustrate the potential impact and applications of our study.

Q3. Simpler baseline that directly utilizes keyword overlap.

Response: Thanks for proposing this simplified approach. Here, we test using the overlap ratio between candidate keywords and query keywords to predict the likelihood of the candidate becoming core citations.

The results prove that the keywords overlap ratio is truly predictive for core citations, achieving performance similar to medium baselines. However, its performance is still far behind that of advanced text-based methods, including our design. This illustrates the advantage and necessity of predicting core citations based on the texts rather than only considering the keywords.

[1] Large teams develop and small teams disrupt science and technology. Nature 2019.
[2] Papers and patents are becoming less disruptive over time. Nature 2023.
[3] The burden of knowledge and the “death of the renaissance man”: Is innovation getting harder? The Review of Economic Studies 2009.
[4] Scientific Paper Recommendation: A Survey. IEEE Access 2019.
[5] Scientific paper recommendation systems: a literature review of recent publications. International Journal on Digital Libraries 2022.
[6] Defining and identifying Sleeping Beauties in science. PNAS 2015.
[7] Modeling citation dynamics of “atypical” articles. Journal of the Association for Information Science and Technology 2018.
[8] New directions in science emerge from disconnection and discord. Journal of Informetrics 2022.

Dear reviewer,

Thanks again for your valuable suggestions. Following your suggestions, we have rephrased the dense terminology and clarified the potential impact of our citation prediction task. Hopefully, our revision can help the readers better understand the contributions and potential applications of our study.

Please let us know if you have any questions. We are looking forward to further discussion.

The authors

Response to reviewer tEdQ

Q1. Missing baselines, e.g., SPECTER 2.0 and SciMult.

Response: Thank you for providing these up-to-date baselines for scientific texts. We test SPECTER 2.0 [1] and SciMult (both vanilla and MoE) [2] on our tasks as suggested. Also, we investigate new baselines for general text embedding to compensate for the existing ones. We include GTE-v1.5, the upgraded version of GTE [3], where the latter one is our previous best baseline. We find the new baselines surpass our existing ones as expected, and still, our method outperforms all baselines. These new baselines provide stronger evidence of the advantage of our method, and they also indicate the possibility of updating the GTE model, which we used for initialization, to stronger pretrained LMs to further improve the performance of our method.

Here, we show the overall performance for your convenience. Please refer to "result.pdf" in the global response for details.

Q2. SciDocs dataset.

Response: Thanks for mentioning SciDocs, which is a widely applied benchmark dataset in scientific text embedding. The major obstacle to testing our method on SciDocs is the lack of ground truth label of core/superficial citation. As reported in the SPECTER paper [4], SciDocs provides approximately 30K papers in total for the citation-prediction task, including 1K queries with a candidate set of 5 cited and 25 uncited papers for each query. However, the cited papers are regarded equally without more detailed information.

In our design, the core citations are identified on the citation network based on subsequent papers citing them. Regretfully, the 30K papers in SciDocs alone are not enough to build a citation network for identifying core citations, beause the citations among them are too sparse. To solve this dilemma, we built a citation network with hundreds of millions of research papers in Microsoft Academic Graph (MAG), identifying core/superficial citations in it. Since there is no direct mapping that bridges SciDocs and MAG, we cannot locate the raw 30K papers in MAG, and thus, we directly sample new queries and candidates from MAG for testing (Section 4.1). We compare our testing dataset and SciDocs from the following aspects:

• Data scale. Our testing dataset is much larger in scale than SciDocs. We include 50K queries with a set of 10 cited (5 core and 5 superficial) papers for each query.
• Field coverage. Like SciDocs, our testing dataset covers multiple scientific fields. We randomly sampled papers in MAG, where the sampled papers covered all 19 fields in MAG.
• Testing format. Our testing uses the format of query-candidates, which is the same as the citation-prediction task in SciDocs.

Considering these, we believe that the testing results on the new dataset we constructed are convincing enough to compensate for SciDocs' absence.

Still, we agree that it is meaningful to use SciDocs to test how various methods perform on the core-citation prediction task we proposed. To enable this in the future, we will work on adding labels of core/superficial to the cited papers provided in SciDocs. We will locate SciDocs's 30K papers in MAG by comprehensively matching the titles, authors, and keywords. Therefore, we can label the citations in SciDocs as core/superficial, providing our task with a more widely acknowledged dataset, as well as contributing an auxiliary applicable scenario to SciDocs.

Q3. Results in each field. & Q4. Statistical significance.

Response: Thanks for these suggestions. Like [5], we show the detailed performance in each field. Also, we conduct statistical significance tests to better compare our model with the strongest baseline/ablation version. We used the two-tailed t-test between the performance of (1) our model VS strongest baseline and (2) full design VS ablation versions. Please refer to "result.pdf" in the global response for details.

The results provide us with a deeper understanding of the performance and validity of our designs, illustrating that:

• Overall performance (our model VS strongest baseline, Table 2 in paper). Our method surpasses the top baselines in all fields ($p<0.01$ in most individual fields, and $p<0.001$ averaging all fields through t-test), demonstrating its general applicability to a wide range of fields.
• Ablation studies (full design VS ablation versions, Table 3 in paper). Generally, all parts of our designs are valid with significance ($p<0.01$ or $p<0.1$ in overall performance through t-test). Moreover, we notice that when focusing on social science papers, which only comprise a small proportion of all papers, Stage 1 of curriculum finetuning is only slightly beneficial. Therefore, when only applying to social science papers, it is an alternative for users to skip Stage 1 if they want to save computational cost with the cost of a slight performance drop. In contrast, when applying to natural science papers, it is necessary to keep Stage 1 for better performance.

[1] SciRepEval: A Multi-Format Benchmark for Scientific Document Representations. EMNLP 2023.
[2] Pre-training Multi-task Contrastive Learning Models for Scientific Literature Understanding. Findings of EMNLP 2023.
[3] Alibaba-NLP/gte-large-en-v1.5. Hugging Face 2024.
[4] SPECTER: Document-level Representation Learning using Citation-informed Transformers. ACL 2020.
[5] The Effect of Metadata on Scientific Literature Tagging: A Cross-Field Cross-Model Study. WWW 2023.

I thank the authors for their detailed responses and additional experiments, which address some of my concerns. After reading the responses and the other reviews, I decided to increase my rating from 6 to 7. Meanwhile, I strongly suggest the authors include the additional results in their revised version.

Dear reviewer,

Thanks again for your insightful suggestions. Following your advice, we have incorporated the up-to-date baselines and clarified all other issues. Hopefully, our revision can help the readers better understand the contributions and potential impact of our study.

Please let us know if you have any questions. We are looking forward to further discussion.

The authors