InterIDEAS: An LLM and Expert-Enhanced Dataset for Philosophical Intertextuality

The main weakness for the paper is the fact that I believe this work does not fit very well with ICLR as a conference overall. I agree that the focus of the paper is creating a dataset, but I believe the way the ICLR community considers "datasets and benchmark" and how this paper presents it are very different from each other. This piece of work would be an excellent contribution to a Digital Humanities (e.g. Computational Humanities Research), Computational Linguistics (Coling or LREC) or Digital Library (JCDL) conference and would have center stage there. I am afraid that it would instead be a mismatch with ICLR and I believe the authors do not argument this strongly enough why the paper should be part of a deep learning conference (instead of a digital humanities or corpus linguistics venue).

1. This paper is submitted to the wrong conference. To my understanding, ICLR is primarily a machine learning conference looking for technical or at least broadly methodological contributions. The current paper proposes a language understanding task and describes a corpus but makes no strong original methodological or technological contribution -- the technologies used for the corpus creation are all pretty well understood. I believe that the paper would be a much better fit for a venue in NLP that values corpora as a contribution, such as the *CL conferences. Even there, the short conference paper format may however be problematic in terms of the details needed.

2. The paper overstates the novelty and relevance of intertextuality. Yes, intertextuality is a major problem in philosophy, but this is not as unique as it sounds in the first paragraph: it is also central to understanding in other areas of the humanities, including literary studies (as is acknowledged in the paper, Section 2). Similarly, I am wondering why the semantics of modifiers are surprising to the authors (l. 52/53) -- this has been known in semantics (and philosophy of language) for a long time. Finally, I cannot easily follow the claims in l. 62-68: it is hard to believe for me that LLMs that read philosophical texts (rather than other texts) would quasi-magically improve their reasoning and sentiment analysis capabilities. This cannot be a serious claim - or maybe I misunderstand its scope.

3. The major contribution of the paper, in my eyes, is the corpus construction, but a number of really fundamental questions remain unanswered, maybe due to the limitations of the paper format; but the usefulness of the corpus remains extremely hard to grasp without answers to these questions. I. What population do the documents with philosophical texts come from? Some kind of corpus? Why is that corpus then by itself not already a good basis for intextuality research? On the other hand, what evidence can you adduce that this population is sufficient for investigations for the research that you envisage? Are key texts guaranteed to be included? What criteria are even important to evaluate such a corpus for these types of research questions? II. By including only texts that are already in digitized form, do you not introduce bias towards more accessible / more established works? What about other forms of bias? Or is bias OK in the sense of a 'guided' corpus construction? III. The same questions, but asked about the InterIDEAS corpus that results from the selection procedure. In addition: what evidence can you adduce that interIDEAS is a representative sample of philosophy research in the sense that observations/analyses on InterIDEAS tell you something true about the body of philosophical research at large? Or if this is not your evaluation criterion, what is it then? IV. What other aspects of quality are relevant when creating a corpus of philosophical writings? How do they compare to "standard" criteria for linguistic or literary corpora? V. In the evaluation in Tab 1, is is rather hard to understand what exactly the task is: is this P and R relative to a gold standard created by human experts, or some abstract gold standard given by the set of all relevant documents from the population, or something else? More information is needed here. Since this is in essence an Information Retrieval task, maybe adopting best practices from IR might be a good idea here.

4. In the evaluation in Section 5.2 (sentiment analysis) -- if I understand this correctly -- models are fine-tuned on InterIDEAS and then evaluated on "reference-attitude pairs from our dataset" (l. 435/436). In other words, the test data is taken from the data used for fine-tuning. No wonder that the models' performance improves with this fine-tuning step: if my understanding is right, this experiment does not tell us a lot about the quality of the corpus. I would recommend to hold-out a test set not used in fine-tuning (ideally even from a specific topic absent from the fine-tuning dataset) for this type of evaluation.

• More a contribution for application of current LLMs to disciplinary work than a core contribution to learning representations (this conference). For this reason, I think the submission is a bit outside the scope of this conference.
• Uses standard prompt engineering techniques with few-shot demonstrations, which has already been previously well-demonstrated, so it does not constitute a core contribution.
• Since the workflow uses humans in the loop, the comparison is not particularly fair. Also, without a clear delineation of the potential novelty to judge the work, it was not clear whether the work wants to be compared with respect to efficacy or not.
• Diagrams have text that is too small to read (e.g., Figures 2 and 3).
• Parts of the evaluation weren't clear. For example, you evaluate against recall but it was not clear how the authors calculated this. Was it done in a pooled evaluation style (a la Cranfield style IR evaluations) or some other form of ground truth?
• As applied to Digital Humanities, it would be good to relate the metadata format to other metadata initiatives (e.g., TEI) common to DH. What do the authoring teams adopt or are influenced by in creating their metadata format and how does it crosswalk to other metadata formats for cross-referencing?
• The sentiment analysis portion of the work did not seem to be very well connected to the motivation of the work.
• The paper wasn't centrally clear whether their workflow or their corpus was the centre of innovation for the paper, leaving it ambiguous and potentially hard to clarify the novelty of their work.

Below, I outline the main weaknesses of the paper.

The paper addresses the identification of citation contexts and the classification of citation intents, topics that are already well-established within the field of Natural Language Processing (NLP) [1]. Unfortunately, the paper does not contribute new insights, as existing NLP frameworks are sufficiently robust to be applied directly to philosophical texts without significant modification.

[1] Measuring the Evolution of a Scientific Field through Citation Frames (Jurgens et al., TACL 2018)