Thank you for your constructive feedback, which has greatly helped improve the draft. Below, we address your concerns and weaknesses (w1-w4).

[w1] Technical Novelty (distinctions from bootEA)

We respectfully point out that the LLM4EA is significantly distinct from bootEA, we outline three major differences between them:

1. Pseudo-label generation. Comparing with the bootEA, our work generates pseudo labels by directly processing semantic information, while bootEA generates pseudo labels based on similarity scores grounded on embeddings trained on pre-aligned labels. LLM4EA exploits the potential of automating the EA task in a label-free fashion.
2. Unsupervised label refinement. Our label refiner is unsupervised, while BootEA relies on the embeddings learned on training labels to compute the label’s confidence. As a result, directly adopting BootEA to this task cannot mitigate the initial noisy annotated pseudo-labels. As we have shown in Table 1 and discussed in line 265-268 in the paper draft, BootEA still suffers from false labels if directly trained on the noisy labels from LLMs.
3. Optimized resource allocation. We introduce an active search policy to improve the utility of the query budget.

[w2 ] Results on OpenEA V1

Thank you for the detailed review and helpful feedback, we will correct the false statement of the dataset version in the revision. To validate the performance on real-world KGs, we attach the experimental results on OpenEA V1 dataset as below. Due to character limit, we only attach results of strong baselines.

As the results show, our model consistently outperforms baselines. Noteworthy that LLM4EA is a general framework and can employ any base EA model such as RDGCN (last row) to perform label-free entity alignment. We kindly argue that our conclusions and claims hold on this more realistic dataset. And we will include the updated results in the draft revision.

[w3] Clarification of baseline setting

Allow us to clarify the confusion about the experimental setting in weakness3.

1. The input of baselines are pseudo labels.

As stated in the experimental setting (line244-245 in the draft), baseline models are also trained on the pseudo-labels generated by the LLMs. This setting ensures fair comparison because the inputs are the same (same query budget to the LLMs).

2. Baselines perform lower than OpenEA's paper for two reasons:

1) the annotated pseudo-labels are noisy and existing methods cannot handle without a pre-refinement of noisy labels before training; and 2) the training label size is smaller ($0.1|\mathcal{E}|$, but in OpenEA's paper it's $0.2|\mathcal{E}|$ ).

[w4] name bias concern

The use of name information is often referred to as name bias. We answer this concern from two aspects:

1. When name information is used: LLM4EA can generate pseudo-labels and enable robust learning. While most existing methods like MultiKE and RDGCN exploit entity names as features, they <u>rely on pre-aligned pairs</u> for training. LLMs can serve as pseudo-label generators.
2. When names are not available: LLM4EA still works. By processing semantics within attributes, LLM4EA can generate effective labels. We empirically evaluate this on D_W_15K_V1 (1st column in the following table), we masked target entity names with IDs to avoid name bias. And compare with MultiKE and RDGCN which both leverage semantic features.

As shown in the above table,

• LLMs serve as an effective annotator for all methods to perform label-free EA, which justifies the above 1st argument.
• LLMs can generate effective labels without name information, which justifies the 2nd claim.

We hope our responses can satisfactorily address your concerns and we would appreciate it if you could consider raising the score.

We respectfully thank you for the helpful feedback that helps improve the draft. We answer the concerns and address weaknesses (w1-w6) as below:

[w1, w4] Reasonability of using LLMs in EA and comparison with recent methods

We sincerely thank for pointing out this subtle point, and we answer this question from following two aspects:

1. Potential in more challenging scenario. Although we perform evaluation in a simple scenatio (entity names are available) in the experiments, the potential of using LLMs in EA tasks is more than that. Specifically, language models can annotate challenging datasets where no entities names are available, by exploiting the semantic information within textual attributes, where existing name-based methods (e.g. LightEA, SEU, BERT-INT) cannot handle. To empirically evaluate this, we entity alignment on D_W_15K_V1 with LLM4EA, where the target entities are IDs to avoid name bias. Results are as below,

   Alignment performance	Annotation precision
   0.472 (hit@1)	301 (true positive)
   0.654 (hit@10)	49 (false postive)
   0.536 (MRR)	1150 (abandoned)

   As shown in the results, LLMs can generate effective pseudo-labels without the need for identical names and get promising results.

2. Scalability to larger KGs. The target of entity alignment is building a larger unified KG. Although existing finetuning-based models (BERT-INT, SDEA, CEA) can also leverage semantic information, our incontext-learning based framework 1) does not require pre-aligned labels to finetune; and 2) is more scalable as it does not require local hardware configuration for LLMs.

we appreciate the listed methods for boosting the comprehensiveness of the discussion of related work, and we will add these results in the draft revision.

[w2 & w3] Results on OpenEA V1

Thank you for detailed reviews, we will correct the false statement about dataset version in the revision. To validate the performance on real-world KGs, we attach the experimental results on OpenEA V1 dataset as below. Due to character limit, we only show the results of strong&new baselines.

As shown by the results, LLM4EA consistently outperform strong baselines. It is noteworthy that LLM4EA is a general framework and can employ any base EA model such as RDGCN (last row) to perform robust label-free entity alignment with LLMs. We argue that the conclusions and claims holds on this sparse dataset. And we will include the updated results in the draft revision.

[w5] Comparison with existing work that addresses noise labels (REA)

The paper referenced in weakness 5 (REA) also addresses noise in the training set. However, our work is significantly distinct from theirs in three key aspects: 1) Technical novelty. The noise detection model in REA is trained as a binary classifier using an adversarial training paradigm, In contrast, our label refiner employs probabilistic reasoning for robust label refinement. 2) Flexibility. LLM4EA can leverage any off-the-shelf EA models as base models without altering the model architecture or training objectives. 3) Superior performance. We empirically tested the performance of REA on annotated pseudo-labels generated by LLMs and present the results below and show the superiority of LLM4EA.

[w6] Low recall/f1 on semantic matching

Thank you for the detailed review and constructive suggestions. We identified that the low recall was due to the word embedding model not generating embeddings for out-of-vocabulary names. We addressed this by replacing it with a pretrained language model (bert-base-uncased) and implementing a filtering process that only considers 1-1 matching. The updated results are as follows:

We hope our response can satisfactorily address your concerns and we would appreciate if you could consider raising the score.

Thank you for your response and your effort to address my concerns. I appreciate your time and effort. I have raised my score to 4. This is because some parts of my concerns aren't fully addressed. Regarding the motivation, I hope you can address these in further responses:

The ID experiment is interesting. Please provide more detailed settings for this experiment. For example, how do you ensure that the IDs are not memorized by the LLMs? Does the accuracy come from the IDs already being memorized by the LLMs as part of the knowledge base? If so, how can we ensure that the LLMs can identify IDs that are not memorized? Wouldn't this memorization lead to more bias than name-bias issues? If we want to use such systems in applications where the resources are so limited that even names are not available, how can we ensure that the LLMs can still identify the IDs? Could you provide a more detailed experiment on this? For example, reassigning the IDs to the entities in the knowledge base and seeing if the LLMs can still identify the IDs. I assume you may need to design specific prompts, for example, showing the neighborhood IDs, to explore the reasoning ability beyond memorization 1. Scalability to larger KGs. Do you have experimental results on larger KGs such as DBP1M? Using an API to query may eliminate the need for local machines, but the API is still costly and there are rate limits. Do you have a cost analysis of using the API compared to deploying a BERT model locally? I would assume that the BERT solution is far more cost-effective and scalable, while still maintaining decent accuracy.

I would advise the authors to compare the recent SOTAs such as LightEA (https://arxiv.org/pdf/2210.10436), both in terms of accuracy and scalability. Some other works like EASY (https://dl.acm.org/doi/abs/10.1145/3404835.3462870) also have some kind of error correction mechanism. It would be interesting to discuss how your work is different from these works.

Thank you for your helpful feedback that helps us improve the draft. We answer concerns and address the three weaknesses (w1, w2, w3) in the following.

[w1] Ablation of LLM model

The choice of LLM model can affect annotation accuracy and cost. We recognized this factor and employed both GPT-3.5 and GPT-4 as annotators, comparing the differences (Table 1 & Figure 2 in the draft) between less powerful and more powerful LLMs. Thank you for the suggestion; we will include experimental results of additional LLMs, especially open-source ones, to complement the empirical analysis.

[w2] Cost measurement

As we have stated in the experimental setting (line244-245), baselines and our framework use the same budget (line239), which is $0.1\times|\mathcal{E}|$. To ensure a fair comparison, we use the same prompt template for all experiments. This also ensures that the actual API cost for baselines and our framework is statistically the same (although a slight variance brought by the different lengths of entity names). Based on our experimental observation, the average API cost for annotating each entity is around 1100 tokens, thus it takes around 1100x(0.1x15000)x0.5/10^6 =0.825 dollars to run each alignment task using gpt-3.5-turbo-0125.

[w3] Improvement of presentation.

Thank you for the detailed review, we will replace the icon in figure1 in the draft revision.

Thank you for your time and effort in reviewing our work. We appreciate your insights and will address these points. Below is a summary of the content of our work.

• Motivations. Our work is motivated by the fact that existing methods heavily rely on accurate seed alignment pairs for training. Annotating such pairs requires substantial cross-domain knowledge, making it very costly. Large Language Models (LLMs) offer new opportunities in automating this task by processing semantic information and generating pseudo-labels.
• Challenges. Employing LLMs for this task is nontrivial because the search space is vast, and LLMs can generate false labels, which can harm the final performance. There is no existing framework to handle these challenges effectively.
• Contributions. As itemized at the end of the introduction section (lines 69-84), our contributions include 1) A novel LLM-based in-context learning framework for label-free entity alignment; 2) An unsupervised label refiner to enable effective training on noisy pseudo-labels; and 3) An active sampling module to maximize the utility of the annotation budget.

We sincerely thank you for your recognition and helpful feedback. We will further polish the paper presentation based on your comments in the revision.