DrugAgent: Multi-Agent Large Language Model-Based Reasoning for Drug-Target Interaction Prediction and Repurposing

1.The contribution of this paper is limited. The proposed model mostly reuses well-known techniques from literature (e.g., DeepPurpose), integrated in a seemingly correct (and mostly non-innovative way).

2.The idea is reasonable and convincing, which seems it focuses more on engineering. The implementation is sensible but does not add much to existing body-of-work in ML/AI. We would have expected to see some more elaborations as concerns the proposed technique.

3.The writing skill is not mature enough, and the relationship between multi-agents system and downstream tasks is not clear. It can also be established by changing to other related tasks such as DDI prediction.

4.The experimental results is not enough to demonstrate the performance of proposed method. As a key problem in drug discovery, many state-of-the-art deep learning methods are proposed to predict DTIs.

• The novelty is very limited. The method (tools design and agent design) is straightforward, the proposed agent is more like a small-scale engineering efforts
• Evaluation is not sound. The scale of 10 test cases is too limited, and there is no evaluation with non-agent baselines (such as existing methods for DTI prediction)
• Related works are missing: many works exist about agents for scientific discovery, it would be nice to reference them and discuss the difference
• Some design choices are lack of sufficient justification or empirical evidence (please see the questions section below)

My main concern with this paper is that I fail to see a major contribution. Let me clarify:

While the overall results seem promising, they indeed stem from combining the hard-coded scripts, mainly from the trained ML model and to a lesser extent from mining a curated KG, rather than performing any (autonomous) reasoning per se. More specifically, it is not clear to me what the contribution of LLMs is specifically - all the interaction with external tools is hard-coded and the combination of the final scores is also pre-defined. The more interesting parts from AI/NLP perspective, the KG agent and the search agent, could have used some level of interaction, in order to find dynamic parameters for scoring the paths and/or search results, respectively, but they do not do so - the weighting of the paths in the KG is hard-coded and the scoring of the google search results is based on keyword matching. Motivation and rationale for choosing these exact weights is missing.

Furthermore, I find it a little odd that the statistical significance results are only reported for comparisons between GPT4 and the full system, comparisons between the full system and the ablated agents are missing, suggesting that they're indeed not significant and performance is likely driven by the ML model predictions (unsurprisingly so).

Minor bits concern exaggerations, for example section 3.2 claims to use ReAct and LEAST-TO-MOST reasoning methods, yet I fail to find their trace in the supplemented code. Furthermore, I am not sure about the use of the term "agent" where all the "agents" do are pre-determined sequences of actions such as calling hard-coded scripts or combining values using a pre-defined formula. Similarly the concluding remark in the introduction is certainly true: "The framework we developed, although initially designed for biological applications, can be adapted to various other fields requiring multi-perspective." - but only if the person adapting the framework is willing to write and integrate the python tools which do all the work.

Overall, I think this could make a nice demonstration paper, but I doubt the contribution is major enough to warrant acceptance at ICLR.

1. Limited baseline comparison: The main evaluation only compares with simple prompting on GPT-4 and model ablations (without AI/KG/Search agent). Given the multi-modal nature of the database (KG and textual literature), comparisons with traditional KG and NLP link prediction methods, as well as RAG models, are notably missing.

I suggest the authors improve the soundness of the paper by including comparisons with 1) traditional ML methods 2) traditional KG methods 3) RAG models 4) other multi-agent systems from the general domain.

2. Questionable "multi-agent" framework:
   • The current design appears to be a simple linear combination of different models' results rather than a true multi-agent system.
   • The role and function of the coordinator agent is unclear.
   • The weights of the linear function are determined practically rather than through principled agent collaboration.

I suggest the authors consider more flexible ways of agent collaboration and think about what is the key advantage of a 'multi-agent' system vs a simple combination of different models' result. For example:

• Each agent could output their reasoning process and confidence score
• The coordinator agent could dynamically decide the final score based on these outputs, instead of using a human-defined linear combination
• Implement inter-agent communication instead of having each agent make predictions solely based on its own database and method

Additionally, we can find form the current discussion of the case study that, the current score from each model is like a black box. Adding intermediate reasoning processes as above would also help humans understand the predictions better.