Causal Impact Index: A Causal Formulation of Citations

• I don't really understand the method. A lot of specific questions are in the next section, but I am missing the big picture. Paper $b$ cites paper $a$. When you find nearest neighbors of paper $b$, do you make sure that none of them also cite paper $a$? Do you make sure they are from the same year? Also how can you tell that they are truly similar enough for the estimation to be legitimate? Is there some theoretical requirement that these nearest neighbors to $b$ are close enough?

• Unfortunately "average 20-30 minutes for each PCI(a, b)" is too slow to be adopted at scale as a substitute for citations or influential citations or other metrics

• Also, it makes sense to compare to other alternatives besides the ones you have already, e.g. https://arxiv.org/abs/2102.03234 and https://arxiv.org/abs/2105.08089

Minor:

• "we consider LLM variants pretrained on large-scale scientific text, such as SciBERT (Beltagy et al., 2019b), SPECTER (Cohan et al., 2020), and MPNet (Song et al., 2020)." These are too small to be called LLMs. Safe to call them LMs.
• Figure 3 should have a log y-axis

The concerned limitation of citation count is not very clear. The terminology used, such as "failing to accurately reflect a paper’s true impact," offers intuitive but vague criticisms of the existing citation-count system. Moreover, the authors do not adequately address how CAUSALCITE overcomes these issues. Explicitly, questions like "Is it feasible to formulate the limitations (especially the bias) of citation count in a causal graph?" and "How does CAUSALCITE solve the claimed limitations?" remain unanswered. Providing such information would help clarify the scope and the innovative aspect of the CAUSALCITE model.

Lack of limitation. Dicussion on limitation is very important especially for a new metric since each metric as its pros and cons. Although the authors did not involve this point in the main text, after rethinking I suggest some easy but common cases where CAUSALCITE could fail. Authors could formulate them as assumptions or reflect them in causal graphs, to clarify the scope of this work. Authors are highly encouraged to debate for their completeness, but an index could not be excellent in every aspect. Identifying and formulating the limitations could largely improve the quality of the paper.

• Data sparsity in high-dimensional spaces. Even though cosine similarity is effective in such spaces, if there are not enough 'near neighbors' for a given paper, the method could fail to find adequate matches. This would result in counterfactual samples that are not truly representative, affecting the validity of the causal inference. Imagine trying to synthesize a counterfactual for a groundbreaking paper on a niche subject; the lack of similar papers would make this task challenging.

• Temporal dynamics. Academic papers are not static entities; they gain citations over time, undergo revisions, and may be commented upon. If the method does not account for these temporal changes, the synthesized counterfactual may be not accurate. For instance, a paper published ten years ago that has been highly cited will have different characteristics than a similar paper published recently, and simply averaging the two could produce misleading results.

• Dimension reduction bias when encoding papers using text embeddings. Although LLMs can capture some textual features, the embeddings are still a reduced representation of the original text. Thus, they might not capture all nuances or specialized details, and numerical results present in the paper, which could be key factors for accessing the paper quality. For example, two papers focusing on a topic, with the same experimental pipeline, could derive different and even contractory results (which is the usual case in some subjects...). Can such difference be reflected by LLM embeddings?

Lack of theoretical backup. The core technical point seems a weighted KNN. Since there are many causal inference methods, such as re-weighting-based, matching-based and representation-based methods, the motivation to select weighted KNN to adjust data should be formulated. As a causality-inspired paper, theoretical formulation and backup is a critical aspect. Is there any theoretical formulation of the advantages of CAUSALCITE over citation count? In which conditions? Moreover, identifiability is a crucial aspect in causality empowered methods. Is CAUSALCITE identifiable? Is there any conditions to support the identifiability of CAUSALCITE?

1. The evaluation counts on the construction of the synthetic dataset, and the reliability of the counterfactual data is not measured or analyzed.

2. The assumption that the confounders are " a paper's publication year, research area, research question, and storyline" is limited. These can be part of the confounder and the storyline is a concept that is hard to measure.

3. To measure the impact of people's work, there might be a need to provide a more rigorous justification, especially when using the LLM for synthetic data generation.