AISciVision: A Framework for Specializing Large Multimodal Models in Scientific Image Classification

Thanks for your candid feedback and suggestions for improvement - we address individual points below:

Bias

While we agree and see your point - that comparing inference data to training data introduces bias to the network - we would argue this is exactly the type of bias we do want to encourage in the LLM - to be more accurate/adapt to the dataset at hand. Bias in general plays an interesting role in machine learning, but we fail to see how this issue is specific to our method.

Domain Specific Tools

You are right in that this is the most important part to get right when working on real world problems - and we are extremely fortunate that our collaborators are domain experts who helped carefully craft the domain tools, and their natural language description to the model. But you are right to be careful about this - we found this to be one of the most critical parts of getting our work to be truly useful as a deployed application! We hope that while these tools were developed for our domains in particular, they can be readily adapted to many of the other domains that require dealing with satellite imagery, or scientific images.

Prediction Cost

Yes this is definitely a trade-off of our method, and improving inference speed and cost is an interesting area of LMM research. However, our domain expert collaborators feel the trade-off is worth it given the higher interpretability.

Questions:

1. “How do you use a hardcoded prompt template for the response?” Sorry, we are not sure exactly what you mean here - the prompts are crafted to be domain specific for each dataset, and the tooling/visrag prompts are hard coded - but the inputs/outputs are dynamic.

2. “How does your model output a probability score indicating its confidence?” We ask the LLM to output an estimate of its probability as a token. This was a design choice on our end and extracting probability estimates from language model outputs is an active area of research.

3. “How do you determine how many conversation turns are enough?” At a high-level, we ask the LLM to do this, by allowing at minimum one turn, maximum four. However in practice, we do heavily encourage three turns - this ultimately comes from working with these datasets, representing a tuneable hyperparameter.

4. “In the paper, you state: "We provide text descriptions of the available tools as prompts to the LMM, and instruct how to request tool use" How do you ensure that the LMM reliably follows your instructions behind the scenes? We know LMMs can sometimes misunderstand prompts, requiring users to read the answer, refine their prompt, and ask again.” We were concerned with this when originally starting on this work - because should the LLM not follow the input/output patterns we set it would actually break the code as the output would not parse - luckily the LLMs seldom ran into this issue.

5. “In the paper, you mention: "You ask the LMM to have 4 conversation turns." How do you determine that this number (4) is appropriate?” Similar to (3) above - this is a hyperparameter decided through experimentation as having a good trade-off. The model can manipulate the data without overloading the framework.

6. “In the paper, you state: "All embeddings for VisRAG are computed via CLIP." How do you ensure that these images are not out of distribution for CLIP?” While CLIP is a substantial foundational model trained on many types of imagery, this is definitely a concern, as out-of-distribution data could limit the efficacy of CLIP to embed them in a meaningful way. However, the success of the kNN model suggests that CLIP embeddings are effective in the datasets we study. This is an excellent point, nonetheless, and we are exploring using some satellite-image-specific embedding models for our future work.

7. “In the paper, you claim that "AISciVision consistently outperforms all baselines on all three datasets." What about the descriptive text your framework returns? Did you conduct any analysis on the quality of those texts?” We do qualitatively get feedback from our domain coauthors about the efficacy and usefulness of the transcript response. Unfortunately, we are still working on a quantitative study based on human interaction, which will take a considerable amount of time.

Thanks for your encouraging feedback and thoughtful questions! We address individual points below:

Novelty

Thank you for this observation. We agree, while the individual components of Visual RAG and tool usage with LLMs have been explored separately, our key contribution lies in their integration and adaptation for scientific domains. We demonstrate that this combination is particularly powerful for specialized scientific tasks where interpretability and domain expertise are crucial. The framework's success across multiple scientific domains (aquaculture, eelgrass disease, solar panels) and its real-world deployment with domain experts provides strong validation of this approach. Furthermore, our analysis shows that the synergy between VisRAG and domain-specific tools enables more robust and interpretable scientific image classification than either component alone could achieve.

Human Evaluation

Thank you for this helpful suggestion - you're right that evaluating the correctness of our model's reasoning would add valuable validation to our work. While our current results demonstrate strong quantitative performance and our domain experts have provided positive qualitative feedback on the reasoning transcripts, a systematic human evaluation would indeed be beneficial. We are currently collecting such evaluations through our deployed application, though this process takes time given the need for expert review. We look forward to sharing these additional insights in future work as they become available.

Questions:

1. “The method says that the most similar positive and negative images are retrieved for a test image. However, in the example transcripts, I can't seem to find where this step is happening. Could you explain how this retrieval is happening through an example?” In the example transcripts (e.g., C.1), we indicate retrieved images to the model with statements like "This is an example with an aquaculture pond" and "Here is an example without an aquaculture pond." Your point raises an interesting direction for improvement - we could more explicitly explain to the model the purpose of these comparative examples, potentially leading to better performance.

2. “The naive baseline of k-nn itself is very strong baseline. Did you find any particular reason for that?” The strong kNN performance can be attributed to its use of CLIP embeddings. CLIP is already a powerful model that has learned a rich embedding space, which enables even simple nearest-neighbor approaches to perform well in this space.

3. “It is said that the evaluation are done in two settings: low label (20%) and full labeled (100%). Where are these labels used? to train CLIP+MLP? or to retrieve +ve and -ve examples during Visual RAG?” The stated percentages of labeled data (20% and 100%) are used both for training the supervised models and for populating the retrieval database used by the VisRAG component.

Thanks for your encouraging review and suggestions for ablation experiments! We address individual points below:

Computational Cost

Indeed, we see this as the main trade-off of our method, as inference is significantly more expensive than traditional machine learning methods. However, the benefits of having interpretable inference output justify this cost - a view confirmed by the domain experts we work with.

Binary Classification

This is a valuable point - the VisRAG component, as implemented, is built specifically for binary classification. However, as noted by reviewer (BMJ1), an interesting reframing of our method would be to consider VisRAG as just another tool - where the model could request similar examples for any specific class. This approach would enable extension to multi-class settings, which we are exploring in ongoing work.

Bias from Supervised Model

This is an important issue, as we pointed out and you note, this was a particular failure case observed for the eelgrass dataset. However, we believe this failure case stems from the fully supervised setting, which underperforms and exhibits more bias than AISciVision. While this is a limitation, it reflects the challenging nature of the dataset for all ML models rather than a specific weakness of our approach.

Questions:

1. “It would be interesting to perform an experiment without supervised model in the toolset, considering that the LMM can be heavily influenced by the model.” This is a helpful suggestion! - and other reviewers also noted we should study removing the other geo-spatial tools. This is addressed in the general comment above, and we performed this ablation experiment (results can be found in Appendix E of the updated paper PDF).

2. “There is no mention of the train test split for each dataset. It can be relevant information” This has been made clearer in the experimental section - thank you for pointing this out.

3. “There is a drop in performance of kNN going from 20% to 100% on Aquaculture dataset. Any explanation for why this is happening?” This is an insightful observation that we overlooked - thank you for noticing! Upon further analysis, we believe the kNN performance drop for Aquaculture is due to the choice of CLIP embeddings, which are not optimized for representing satellite images (the CLIP paper authors note this limitation, see Section 3.1.5 of https://arxiv.org/pdf/2103.00020). One possible explanation is that CLIP embeddings don't effectively distinguish between satellite images. In the 100% vs 20% setting, we believe this limitation causes the kNN classifier to retrieve training images that are less similar to the test image. This suggests that effective dataset embeddings are crucial for AISciVision's VisRAG component—an avenue we plan to explore in future work.

Thank you for your thorough review and insightful comments. We address individual points below:

Motivation/significance of transcript generation for scientific classification

We are expanding the motivation section in the revised manuscript in conjunction with our ecologist collaborators to better articulate the significance of the selected scientific classification tasks and their broader scientific impact. For example, accurate automated aquaculture pond detection in the Amazon is crucial for monitoring illegal deforestation, water quality impacts from runoff, and land usage for estimating carbon emissions, where misclassification could lead to overlooked environmental violations or wasted conservation resources. Similarly, early detection of eelgrass wasting disease helps prevent devastating coastal ecosystem collapse, as these plants are fundamental to marine habitat health and carbon sequestration.

Further, we will emphasize that generating reasoning transcripts is particularly critical for our use cases. Ecologists working in challenging regions, such as the Amazon, rely on predictions that mimic their own decision-making process, by following a checklist and weighing multiple criteria. These transparent transcripts not only provide a detailed rationale for each prediction but also build trust in the system. Importantly, this framework is actively deployed and in real-world use, where interpretability and trust are essential for adoption.

Stronger baselines, other LMMs, and fair comparisons

Thank you for these suggestions! We address these points in detail in the general comment - but we have run ablation experiments to cover all of these settings (2 more LMMs, a Resnet50 baseline, and a same-context comparison), and we believe the results have really helped confirm the necessity of all interoperating components.

More details on tooling

We will add a discussion of our domain-specific tools to the main paper, highlighting that they arise from collaboration with domain experts and can be generally applicable to other domains. We are working on fitting that discussion into the main paper. Furthermore, we conduct an ablation experiment with tool choices in Appendix E (see general comment).

Questions:

1. We found that this approach lets the model compare and contrast both the similar and dissimilar classes, mimicking how a human would approach the task. We now include an ablation experiment testing this (more details in the general comment above).

2. While GPT-4o with VisRAG alone shows strong performance, the domain-specific tools serve a crucial role in providing interpretability and verification - they allow the model to actively test its hypotheses and generate detailed reasoning transcripts that explain how each tool operation contributed to the final classification decision, making the system more transparent and trustworthy for scientific applications.

3. We conduct an ablation experiment with tool choices in Appendix E (see general comment) for the aquaculture dataset, comparing geospatial tools with the general image enhancement tools.

4. This is a great question and, unfortunately, one that is difficult to answer. The timing of the photos can significantly affect the results, as ponds can appear or disappear over time. We are working with domain experts to address this issue by incorporating other non-public datasets, but for now, it remains a source of systemic noise in our dataset.

5. The prompting was crafted by domain experts to most accurately reflect how they would teach or prompt another human to perform the task - so they are domain-specific.

6. This is an area we are actively studying. Indeed, selecting three tools was a hyperparameter we set, as we found this provided a good balance between allowing the model to explore more data without overloading it.

7. This is a really interesting comment. It comes down to system design - we effectively require the model to use the VisRAG tool because we see it as a very powerful component. However, you raise a great point: it would be interesting to study a design where everything is treated as a tool, letting the model freely decide its use.

8. While both the Eelgrass and Solar datasets leverage similar image enhancement tools (contrast, brightness, edge detection), we classify these as domain-specific because they were selected based on expert consultation and common analysis practices in their respective fields - plant pathology experts commonly use these visual adjustments to identify disease patterns, while remote sensing experts use similar enhancements to detect built structures in satellite imagery.

9. As of now, because we are working specifically with domain experts in the Amazon, they always have latitude/longitude metadata available. For other settings where this metadata is not available, we could craft alternative tooling.

I would like to thank the authors for their detailed responses and for adding ablation studies, which have addressed most of the questions raised in my initial review. However, I have a few additional clarification questions for further discussion:

C2: Domain-specific tools serve a crucial role in providing interpretability and verification.

Question: Given that GPT-4o + VisRAG contributes most significantly to the performance gain, are the domain-specific tools primarily serving an interpretability role? Additionally, since the LMM outputs already provide interpretability in explaining classification decisions at each turn even after VisRAG, why are the domain-specific tools necessary for interoperability?

C5: The prompting was crafted by domain experts.

Question: For clarification, the initial prompts differ across datasets. For the Aquaculture and Solar datasets, the initial user prompt is: “This is an example of a satellite image with an <dataset_class>. Describe what you see, noting the characteristics that identify it as an <dataset_class>.” Whereas, for the Eelgrass dataset, the initial user prompt is:: “I will start by first showing you an example image, that is visually similar to the image we want to classify, that does have eelgrass wasting disease ….” Could you explain why different prompts were used for these datasets?

C8: Tools were selected based on expert consultation and common analysis practices.

Question: The domains of eelgrass and solar panels are quite distinct, yet the same set of tools is provided as a choice for both. Could you clarify how these are referred to as “domain expert tools”? Moreover, how do you determine the expected outcomes of these domain-specific tools? Are there studies or evidence showing that domain experts are already employing these kinds of tools in their workflows?

Additional Questions

1. AISciVision shows performance gains only on the Aquaculture dataset. For the Solar dataset, CLIP+MLP / ResNet outperforms or matches the AISciVision model across 5 out of 6 metrics. Similarly, for the Eelgrass dataset, the baselines outperform AISciVision on 4 out of 6 metrics, with comparable performance on the other two. If AISciVision does not provide significant classification performance gains, why not position the paper as an interpretable tool, similar to techniques like CAM [C] or Grad-CAM [D]?
2. What unique advantage does AISciVision offer over existing interpretability techniques, such as CAM [C] or Grad-CAM[D], which are already well-established for classification tasks?

References :

[C] Zhou, Bolei, et al. "Learning deep features for discriminative localization." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.

[D] Selvaraju, Ramprasaath R., et al. "Grad-CAM: visual explanations from deep networks via gradient-based localization." International journal of computer vision 128 (2020): 336-359.

Thank you for the additional questions! We are glad that our response and new experiments address your initial questions. We address individual points below:

Domain-specific tools and interpretability

We respectfully disagree with the characterization that domain-specific tools primarily serve an interpretability role. Our evidence demonstrates their substantial functional value. The ablation results in Table 2 definitively show that GPT-4o+Tools outperforms GPT4o+VisRAG on the aquaculture dataset in both Accuracy and F1 score, while the full AISciVision framework (GPT4o+VisRAG+Tools) consistently achieves the highest metrics across datasets. This indicates that tools and VisRAG serve distinct, complementary functions rather than tools being merely interpretability aids. The tools enable precise specialized analytical operations that neither the LMM nor VisRAG can perform independently, such as edge detection, and further analysis that are essential for accurate classification. Importantly, these tools don't just explain decisions - they fundamentally shape how decisions are made by enabling the LMM to perform the same analytical steps that domain experts use to reach their conclusions.

The tools serve two equally important functions: enabling accurate classification through domain-specific analysis capabilities while simultaneously providing clear interpretability of the decision-making process.The fusion of VisRAG's broad visual understanding with the tools' specialized analytical functions creates a system that better replicates expert workflows, leading to both better performance and more trustworthy results.

Slight prompt difference in datasets

We appreciate you bringing this to our attention, and sorry we missed your initial point! The prompt difference between datasets is an unintentional artifact from our prompt refinement process during development, where we failed to standardize the prompts across all datasets in our final implementation. Upon thorough review of our experimental results and the tool-call processes, which use consistent prompting across datasets, we are confident that this prompt variation does not materially impact our reported findings or conclusions. Nevertheless, we acknowledge this inconsistency and have updated our codebase to use standardized prompts across all datasets for better reproducibility and clarity. Thank you for catching this!

Tools selected by consulting experts

Thank you for this important question about domain specificity. The tools have varying levels of domain alignment across our datasets. For aquaculture and eelgrass, our tool selection was directly informed by working with our ecologist collaborators who use similar analytical approaches in their workflow. The solar panel dataset was included as an additional test case to evaluate our framework's generalizability to other remote sensing classification tasks, where we just used similar image processing techniques as tools.

While our tool selection shows promise across these different domains, we acknowledge that future work would benefit from deeper domain expert consultation for each specific use case to further optimize tool selection and integration.