Can LLMs Reason Over Non-Text Modalities in a Training-Free Manner? A Case Study with In-Context Representation Learning

We sincerely thank Reviewer nfkW for the valuable comments. For the concerns and questions, we provide the following responses.

---

W1: Missing Lightly-Trained Baseline for Trade-off Analysis

Thank you for the insightful comment. To better highlight the differences between training-free and conventional training-based methods, we designed experiments from two perspectives and conducted a comprehensive analysis.

First, considering that ICRL's performance bottleneck largely stems from the LLM's unfamiliarity with embedding-style inputs and its lack of molecular knowledge, we implemented two strategies for training the projector connecting the FM and the LLM. The first is inspired by CLIP, aligning the FM-projected embedding with the LLM's intermediate hidden states after processing the corresponding SMILES input. The second strategy involves pretraining the projector on a molecular captioning dataset.

While both methods improved the LLM's ability to interpret FM-derived inputs, they still underperformed compared to our training-free ICRL approach on downstream molecular prediction tasks (detailed in Reviewer J6ma Q2). Furthermore, we surveyed relevant literature and included comparisons with a diverse set of training-based methods:
-- instruction finetuning with Q-LoRA,
-- fully supervised pretraining + finetuning, and
-- unsupervised pretraining followed by supervised finetuning.

We conducted a detailed comparison in terms of training paradigms, required computational resources, training time, and performance on two benchmark datasets. We found that ICRL achieves comparable or even better results than these methods at the same inference cost—all without any training (see Reviewer S7er W1 for details).

In other words, while lightweight tuning can indeed improve performance to some degree, its effectiveness is limited when the base model is not pretrained on molecular data. In contrast, ICRL allows a general-purpose LLM to surpass these training-based methods with only ~2 seconds of test-time cost, demonstrating both the practicality and effectiveness of our approach.

---

W2: Narrow Modalities and Task Types

Thank you for the insightful comment.

Due to space constraints, we focused our main analysis on molecular modality. However, we have also included additional results for proteins (Appendix E.3) as well as vision and audio (Appendix E.6), and we are committed to providing the same level of in-depth analysis for these modalities in future versions.

Importantly, the insights gained from our analysis on small molecules generalize well to other modalities. For example, as discussed in Section 5, high intra-set similarity in FM representations can hinder ICRL performance. Our follow-up experiments on the protein modality confirm this: by replacing the FM encoder with one that produces more distinguishable representations, we observe significant improvements in ICRL performance (see Reviewer J6ma Q1).

We also acknowledge that our initial evaluation primarily involved regression tasks and a few classification tasks. To address this, we have added more classification tasks (e.g., molecular QA) and generation tasks (e.g., molecular captioning) to more comprehensively evaluate ICRL's effectiveness. Specifically, on the QA benchmark, even embedding-level injection alone significantly outperforms both fine-tuned general LLMs and ICL. For captioning, although there is still a performance gap relative to expert models, representations injected via OT-PCA consistently outperform SMILES-based text examples and help the LLM better interpret unfamiliar inputs (detailed in Reviewer aUho W2).

---

Q1: Unclear Motivation Over Fine-Tuning

Thank you for pointing this out. We acknowledge that the original manuscript did not elaborate on the motivation in sufficient depth, and we will revise the discussion accordingly in the next version.

First, we would like to clarify that embedding-level injection methods such as OT-PCA, similar to lightweight training, require only a one-time computation of alignment parameters. At inference time, applying the projection amounts to a single matrix multiplication—even when using over a hundred examples, the total cost is well under one second, and can be ignored in practice. In fact, computing the OT alignment parameters for ICRL takes less than 2 seconds on CPU, which is orders of magnitude faster than even the most lightweight fine-tuning setups that typically require at least several hours on 2–4 GPUs.

More importantly, our latest experiments demonstrate that ICRL achieves performance comparable to, or even exceeding, lightweight training approaches across multiple datasets, modalities, and task types (see Reviewer S7er W2, Reviewer J6ma Q2). While large-scale pretraining + fine-tuning paradigms can yield strong performance, they often demand days or weeks of training.

Crucially, both lightweight and large-scale training methods are inherently task-, FM-, and LLM-specific—any update in task formulation, foundation model, or dataset requires costly re-training or re-tuning from scratch. In contrast, ICRL can be seamlessly applied to new modalities, and benefits directly from improvements in both FM and LLM backbones (see Reviewer J6ma Q1, Appendix E.5), all with a negligible CPU-only alignment step. Despite its minimal cost, ICRL achieves comparable or better results than many training-based approaches.

Compared to traditional many-shot ICL, ICRL also offers superior generalization. Even when the LLM lacks domain-specific knowledge, ICRL can leverage FM-derived signals to outperform standard ICL (see Reviewer aUho W2). More importantly, for modalities like vision and audio, general-purpose text-based LLMs cannot handle the inputs at all, while ICRL remains effective—bridging these gaps (see Appendix E.6).

In summary, ICRL offers a highly cost-effective, training-free alternative to fine-tuning, providing a practical way to leverage the growing pool of open-source LLM and FM weights in the AI community. We believe it opens a promising direction for multimodal research that challenges the conventional reliance on large-scale training.

---

As reflected in our title, ICRL serves as a case study in our broader effort to explore how LLMs can reason over non-text modalities in a training-free manner. We sincerely thank all reviewers for their thoughtful feedback and will incorporate the suggestions and insights from this rebuttal into the next revision and future work.

We sincerely thank Reviewer aUho for the valuable comments. For the concerns and questions, we provide the following responses.

---

W1: Using Fine-Tuned Molecular LLMs as an upper bound

Thank you for your insightful comment.

After carefully reviewing the three referenced papers [1, 2, 3], we would like to clarify that their core methodology involves appending 2D or 3D molecular representations—via trainable modules—to 1D SMILES embeddings, thereby enhancing the LLM's ability to interpret molecules. In contrast, our current work focuses specifically on analyzing the 1D SMILES input scenario, and thus the pretrained molecular LLM embeddings from these models are not directly comparable to our setup.

That said, we truly appreciate the reviewer's suggestion of using trained molecular LLMs capable of processing higher-dimensional inputs as an upper-bound reference for ICRL. In this spirit, we have designed two experimental strategies:

-- Concatenated embeddings: Combining embeddings from multiple molecular views (1D/2D/3D) into a single vector as the FM representation.

-- Separated embeddings: Treating the additional dimensions as supplementary context to the original 1D SMILES-based input.

Due to time and computational resource constraints, we were unable to fully implement and analyze these experiments during the rebuttal period. However, we are committed to incorporating these directions—including references to related work—in a future version of the paper.

It is also worth noting that while recent molecular LLMs are trained to better understand molecular structures, this does not imply that they directly perform well on downstream tasks such as property prediction. For example, we experimented with training a projector to connect FM embeddings with the LLM, which resulted in stronger molecular captioning performance—but significantly worse performance on property prediction compared to our training-free OT-PCA approach (see Reviewer J6ma Q2). Similarly, our literature review shows that pretraining strategies focused on molecular understanding, and even lightweight fine-tuning methods, often underperform compared to ICRL in downstream tasks (see Reviewer S7er Q1).

---

W2: Limited Task Diversity such as molecular captioning and QA benchmark.

Thank you for your insightful comment.

We agree that broader task coverage is important for evaluating the generality of ICRL. In response, we have incorporated additional evaluations on the molecular QA benchmark [4] and molecular captioning [5] tasks. These new results help us better understand the capabilities and behavior of ICRL under different task types. We are committed to including these findings and discussions in a future version of the paper.

Setup:

-- All methods use LLaMA-3.1-8B-Instruct as the base model. For the ChEBI-20 dataset, in-context examples are randomly sampled from the training set at each run. For MoleculeQA, the examples are uniformly sampled from the training set during each evaluation. To ensure fairness, we disabled batch querying in our implementation, as it was not used in baseline methods. Other settings follow the original paper.

-- For the captioning task, we report BLEU-4, ROUGE-1, and ROUGE-L as evaluation metrics. For the QA task, we use accuracy. Baseline results in the table are taken from [4] Table 8 and [5] Table 1, respectively.

Analysis:

-- As shown in Table 1, the training-free ICRL approach outperforms both standard ICL and fine-tuned general-purpose LLMs on the QA task. This result demonstrates that, although there remains a performance gap compared to expert models, the processed FM representations can indeed be interpreted and partially utilized by a text-based LLM, even one that has never encountered representation-type inputs. Interestingly, in this task, embedding-level injection alone performs better than when combined with text features, suggesting that the injected representation serves as a more effective signal than the original SMILES strings in helping the LLM understand molecular structure.

-- As shown in Table 2, despite the general LLM lacking inherent capabilities in molecular captioning, the embedding-level OT-PCA approach still outperforms standard ICL and effectively enhances the model's ability to generate captions from text features. This again supports the effectiveness and generalizability of our method, even on generative tasks.

Conclusion:

Overall, while ICRL performance is inherently influenced by the capabilities of the underlying FM and LLM on specific tasks, our experiments on the QA benchmark and molecular captioning clearly show that the injected representations can be meaningfully used by general-purpose LLMs. Moreover, ICRL significantly outperforms lightweight tuning baselines, which is consistent with our findings on regression tasks (see Reviewer S7er Q1).

---

W3: Inadequate Metrics — Recommend Reporting RMSE instead of Pearson's r

Thank you for your comment.

We would like to clarify that for all regression experiments in the paper, we evaluated performance using RMSE, Pearson's r, and Spearman's ρ. The conclusions we reported are consistent across all three metrics (see Line 210 and Footnote 2). Due to space limitations and the growing trend in regression evaluation to include correlation coefficients for a more comprehensive view [6,7], we chose to report Pearson's r in some cases. Full results using all metrics, along with relevant discussions, are provided in Appendix F.2.4.

---

Q1: How to select the in-context examples

Thank you for pointing this out.

In all our regression experiments, we selected in-context examples via stratified sampling from the training set. Specifically, for each run, we partition the training set into equal-sized bins based on the number of examples required, and then randomly sample one example from each bin. We will include this implementation detail in the revised version of the paper.

---

[1] Towards 3d molecule-text interpretation in language models. ICLR 2024.

[2] LLaMo: Large Language Model-based Molecular Graph Assistant. NeurIPS 2024.

[3] Mol-llama: Towards general understanding of molecules in large molecular language model. ArXiv 2025.

[4] Moleculeqa: A dataset to evaluate factual accuracy in molecular comprehension. EMNLP 2024.

[5] Translation between molecules and natural language. EMNLP 2022.

[6] Regression Transformer enables concurrent sequence regression and generation for molecular language modelling. Nature Machine Intelligence 2023.

[7] Gradient Aligned Regression via Pairwise Losses. ICML 2025.

---

As reflected in our title, ICRL serves as a case study in our broader effort to explore how LLMs can reason over non-text modalities in a training-free manner. We sincerely thank all reviewers for their thoughtful feedback and will incorporate the suggestions and insights from this rebuttal into the next revision and future work.

Dear Reviewer aUho,

Thanks so much again for the time and effort in our work. Considering the limited time available and to save the reviewer's time, we summarized our responses below.

[W1: Using Fine-Tuned Molecular LLMs as an Upper Bound]

Response: Our current work specifically targets the 1D SMILES input scenario, making direct comparisons with pretrained molecular LLM embeddings (which typically use 2D/3D inputs) challenging. However, we appreciate your suggestion and have proposed two experimental strategies (concatenated and separated embeddings) to use molecular LLMs as an upper bound in future work.

Also, preliminary analysis indicates that molecular understanding gained from such LLMs does not necessarily improve downstream task performance compared to our training-free ICRL.

We will explicitly include these experiments and findings in future versions of the manuscript.

[W2: Limited Task Diversity such as Molecular Captioning and QA Benchmarks]

Response: We expanded our evaluation to include molecular QA and captioning tasks, demonstrating that ICRL significantly outperforms standard ICL and fine-tuned general-purpose LLMs. Specifically, our embedding-level OT-PCA approach shows meaningful improvements in both QA and caption generation tasks, supporting ICRL's general effectiveness.

These findings will be integrated into future revisions of the paper.

[W3: Inadequate Metrics—Recommend Reporting RMSE instead of Pearson's r]

Response: We clarify that our evaluation is comprehensive, employing RMSE, Pearson's r, and Spearman's ρ, with consistent conclusions across these different metrics. The choice of Pearson's r for concise reporting aligns with practices adopted in recent regression evaluation studies.

Complete results with detailed discussions are provided in Appendix F.2.4. We will explicitly highlight this multi-metric approach in the revised manuscript.

[Q1: How to Select the In-Context Examples]

Response: In all regression experiments, we selected in-context examples via stratified sampling from the training set. We partition the training data into bins and sample one example from each, ensuring representative selection.

We will explicitly include this detail in future versions of the manuscript.

Since the discussion stage is already halfway through, may I know if our rebuttal addresses the concerns? If there are further concerns or questions, we are more than happy to address them. Thanks again for taking the time to review our work and provide insightful comments.

Best regards,

Authors

We sincerely thank Reviewer AQ11 for the valuable comments. For the concerns and questions, we provide the following responses.

W1: Questionable Value of Training-Free Setting

Thank you for your comment.

While many cross-modality adapters are indeed lightweight, their effectiveness heavily depends on both the base LLM and the downstream task. For example, in molecular property prediction, ICRL achieves comparable or even better performance than lightweight fine-tuning or short-duration pretraining + fine-tuning methods—all at similar inference cost and in a training-free manner (see Reviewer S7er W2, Reviewer J6ma Q2).

Although simple lightweight tuning can improve downstream performance, its effect is limited when the base model has not been pretrained on molecular corpora. In contrast, ICRL enables a general-purpose LLM to surpass these methods with only ~2 seconds of test-time computation per sample, showcasing both the efficiency and effectiveness of the approach.

More importantly, while fine-tuning is relatively inexpensive compared to full pretraining, it still requires substantial GPU resources and hours to weeks of training time whenever a new downstream task, FM, base LLM, or dataset is introduced. In contrast, ICRL can be seamlessly applied to any new modality and benefits directly from improvements in both the FM and LLM (see Reviewer J6ma Q1, Appendix E.5). The only additional cost is a CPU-level alignment step, which takes under 2 seconds, yet yields performance comparable to or even exceeding many traditional approaches.

Additionally, for closed-source LLMs accessed via API, we can still use open-source tokenizers to decode pre-aligned embeddings into text form and feed them as inputs. We plan to include experiments in this direction in future work.

---

W2: Chemistry-Only Validation isn't a Good Fit for NeurIPS

Thank you for your comment.

Our experiments go beyond small molecules and include multiple modalities such as proteins (Lines 334–340, Appendix E.3), vision, and audio (Lines 341–353, Appendix E.6). We have also added question answering and generation tasks (see Reviewer aUho W2), demonstrating both the effectiveness and generality of our approach across domains.

Moreover, evaluating a method within the chemistry domain does not inherently disqualify it from NeurIPS. Numerous impactful works have been accepted at NeurIPS while focusing exclusively on molecular or chemistry-related problems [1, 2, 3].

---

W3: Issues about Figure 1 Caption and two Level injection methods.

Thank you for your comment.

We would like to clarify that Figure 1 illustrates the differences between (i) traditional ICL methods, (ii) time-consuming, projector-trained multimodal LLMs, and (iii) our training-free pipeline. Due to space limitations and the intuitive nature of the comparison, we placed this figure in the Introduction (Lines 36–47), where it is explained in detail.

As reflected in the paper title, our goal is to explore how LLMs can reason over non-text modalities in a training-free manner. The proposed text-level and embedding-level strategies represent two distinct approaches to this problem. Text-level injection is more intuitive but suffers from limitations such as context window constraints. As a result, we started our investigation with text-level methods and subsequently extended our analysis to embedding-level injection (see Lines 48–57 and 116–120 for detailed discussion).

---

Q1: Issues about Abstract.

Thank you for pointing this out. We will revise the abstract in the next version to include a clearer articulation of the research problem and the corresponding conclusions.

---

Q2: Issues about performance.

Thank you for your comment.

We would like to clarify that the sentence in the paper—"our primary objective is not to outperform ICL but to investigate the feasibility of adaptively integrating non-text FM representations into a text-based LLM in a training-free manner"—is not meant to suggest that performance improvements are out of reach. Rather, it emphasizes our focus on methodological contribution.

In practice, LLMs are not inherently capable of interpreting FM-derived embeddings, and even domain-specific fine-tuning of projectors often fails to bridge this gap effectively (see Reviewer J6ma Q2). As a result, embedding-only injection can occasionally underperform compared to ICL. However, when used in conjunction with textual examples, ICRL consistently outperforms standard ICL, with improvements reaching up to 16.6% (Lines 223–233).

Moreover, in scenarios where the LLM lacks sufficient internal knowledge (Lines 235–244, Appendix E.5), and in certain domains such as proteins (Reviewer J6ma Q1) and molecular QA (Reviewer aUho W2), even embedding-only injection achieves better performance than ICL, further underscoring the effectiveness and practical value of our method.

---

Q3: Issues about how those projected vectors serve as input to the LLM

Thank you for your comment.

As indicated by our title, the goal of this work is to explore training-free methods for enabling text-based LLMs to understand information from non-text modalities through FM-derived representations. To this end, we propose two complementary strategies for injecting FM features into LLMs:

Text-level injection: The projected FM representation is serialized into a string and fed into the model as part of the text input (Lines 48–52, 107–114).

Embedding-level injection: The projected FM representation is directly concatenated with the token embeddings of other text inputs (Lines 52–57, 116–159).

We further compare these two strategies under different conditions—e.g., whether the FM representations are used alone or together with original textual features such as SMILES (Lines 208–280)—and analyze the underlying mechanisms behind their performance differences (Lines 281–322).

In summary, this work investigates whether there exist lower-cost alternatives to the conventional pretraining–fine-tuning paradigm that can still allow LLMs to interpret and utilize non-text FM representations. To this end, we present both text-level and embedding-level injection strategies. The embedding-level approach, in particular, not only enhances ICL when combined with textual examples, but also achieves performance comparable to or better than time-consuming trained models when used on its own (see Reviewer S7er W1, Reviewer J6ma Q1, Reviewer aUho W2).

---

[1] GIMLET: A Unified Graph-Text Model for Instruction-Based Molecule Zero-Shot Learning. NeurIPS 2023.

[2] Self-Supervised Graph Transformer on Large-Scale Molecular Data. NeurIPS 2020.

[3] DrugCLIP: Contrastive Protein-Molecule Representation Learning for Virtual Screening. NeurIPS 2023.

---

As reflected in our title, ICRL serves as a case study in our broader effort to explore how LLMs can reason over non-text modalities in a training-free manner. We sincerely thank all reviewers for their thoughtful feedback and will incorporate the suggestions and insights from this rebuttal into the next revision and future work.

We sincerely thank Reviewer J6ma for the valuable comments. For the concerns and questions, we provide the following responses.

Q1: More Diverse Set of Evaluations

Thank you for your insightful comment.

While the experiments in the main paper focus on molecular tasks, this choice was made to enable deeper analysis of ICRL's underlying mechanisms under well-controlled conditions. Other modalities—including protein, vision, and audio—were also explored, but due to space limitations, these results were placed in the appendix. In addition, we have included more types of tasks in our evaluations, such as QA and generation (see Reviewer aUho W2), as well as experiments on lightweight projector training strategies (see Q2). We plan to further extend ICRL to a broader range of modalities and task types in future work.

Notably, the key conclusions drawn from the molecular experiments—such as the impact of representation similarity—are modality-agnostic. Specifically, we observed that when using embedding-level representations alone, the intra-dataset similarity of FM representations significantly affects downstream performance. In our earlier appendix experiments, ICL and ICRL performed poorly on protein property prediction tasks, largely because protein datasets often focus on a specific protein function and thus contain sequences that differ by only a few amino acids—leading to extremely high representation similarity.

In our latest experiments, we addressed this issue by replacing the FM encoder with one that produces more diverse representations, and the results below further validate the generalizability of our findings across different modalities.

Setup:

-- Both tables use LLaMA-3.1-8B-Instruct for ICRL implementation, with RMSE as the evaluation metric.

-- Few-shot examples are selected via stratified sampling (20 per dataset); all other settings follow the main paper.

-- Table 1 reports results using ESM2 [3] as the FM encoder, which yields highly similar protein representations (average similarity ~0.98).

-- Table 2 uses ProtBert [4], which produces more distinguishable embeddings (average similarity ~0.92), leading to improved downstream performance.

Analysis:

-- As shown in Table 1, using ESM as the FM yields limited performance improvements—occasionally surpassing ICL but generally underperforming. In contrast, with ProtBert, which provides more diverse embeddings, even using embedding-level representations alone can achieve performance comparable to or significantly better than ICL.

Conclusion:

While most experiments in the main paper focus on small-molecule datasets, the key insights derived from them generalize well across modalities and datasets.

---

Q2: Discussion on understanding Inconsistent Results

Thank you for your comment.

To better understand the inconsistency in ICRL performance, we explored whether training-based strategies could mitigate the model's unfamiliarity with representation-based inputs. While concurrent work [1] demonstrates that projector pretraining can help LLMs interpret such inputs in language tasks, we found that even with explicit projector pretraining, these strategies still underperform in molecular property prediction compared to our training-free approach.

As analyzed in Section 4 of the paper, ICRL performance is influenced by several factors, with the most critical being the characteristics of the LLM used during inference. Specifically, the inconsistency in performance—such as the inability of embedding-level representations to consistently outperform ICL or the varying effects depending on whether textual examples are included—stems from the model's unfamiliarity with representation-based inputs, which are absent during pretraining. While the LLM also lacks domain knowledge about small molecules, it remains highly attuned to textual inputs, making it easier to interpret and utilize them.

Our work focuses on helping the model understand and utilize unseen input types in a training-free manner, allowing it to benefit from expert FM representations without requiring fine-tuning. Traditional methods can achieve similar goals but often rely on extensive pretraining on domain-specific datasets (see Reviewer S7er W1). For example, concurrent work [1] pretrains a projector using molecular captioning data to help LLMs interpret embedding-type inputs. However, this approach requires ~15 hours of training and does not generalize well to downstream tasks—ultimately performing worse than our training-free method.

To further explore this issue, we designed a contrastive training strategy inspired by CLIP [2]. It aligns LLM hidden states by minimizing the distance between the final token representations when receiving molecular inputs either in SMILES form or as projected embeddings. This strategy can also be combined with pretraining to enhance the model's ability to interpret representation-based inputs.

Setup:

-- All methods use LLaMA-3.1-8B-Instruct as the base model, with a single-layer linear projector of hidden size 4096, consistent with [1].

-- The pretraining strategy from [1] uses the Language + Molecules-24 (LPM24) dataset [5]. Our contrastive learning strategy is trained on the training split of each evaluation dataset and evaluated on the LPM24 test set for molecular caption generation.

-- Regression tasks are evaluated on Solubility_AqSolDB and ESOL using RMSE. Other experimental settings follow those in the main paper.

Analysis:

-- As shown in the table, even with projector training designed to help the LLM better interpret representation-based inputs, the performance on regression tasks remains poor—significantly worse than that of the training-free ICRL approach.

-- Improved molecular understanding or caption generation ability in LLMs does not necessarily translate to better performance on regression tasks, which is consistent with our findings in Reviewer S7er W1.

Conclusion:

The inconsistent results of ICRL across settings stem from the LLM's inherent difficulty in interpreting and utilizing representation-based examples. While traditional methods rely on time- and resource-intensive training to mitigate this, such familiarity does not guarantee improved downstream performance. In contrast, ICRL enables the LLM to acquire a certain level of understanding of FM representations at test time, offering a lightweight and effective alternative.

---

[1] Vector-ICL: In-context Learning with Continuous Vector Representations. ICLR 2025.

[2] Learning Transferable Visual Models From Natural Language Supervision. ICML 2021.

[3] Evolutionary-scale prediction of atomic-level protein structure with a language model. Proceedings of the National Academy of Sciences 2021.

[4] ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning. IEEE Transactions on Pattern Analysis and Machine Intelligence 2022.

[5] L+M-24: Building a Dataset for Language + Molecules. ACL 2024.

---

As reflected in our title, ICRL serves as a case study in our broader effort to explore how LLMs can reason over non-text modalities in a training-free manner. We sincerely thank all reviewers for their thoughtful feedback and will incorporate the suggestions and insights from this rebuttal into the next revision and future work.

Dear Reviewer J6ma,

Thanks so much again for the time and effort in our work. Considering the limited time available and to save the reviewer's time, we summarized our responses below.

[Q1: More Diverse Set of Evaluations]

Response: The experiments in the main paper primarily focus on molecular tasks for deeper analysis under controlled conditions. However, we have explored other modalities such as protein, vision, and audio, detailed in the appendix due to space constraints. Additionally, we evaluated more task types, including QA and generation, and conducted further experiments with lightweight projector training strategies. Our latest results confirm that key insights, such as the impact of representation similarity, generalize across different modalities.

Specifically, embedding-level representations using diverse FM encoders significantly outperform ICL alone, while highly similar embeddings offer limited gains. Thus, our primary conclusions regarding representation similarity and model performance are modality-agnostic and widely applicable.

[Q2: Discussion on Understanding Inconsistent Results]

Response: To address inconsistencies in ICRL performance, we explored training-based strategies for projectors. However, even explicit projector pretraining (via molecular captioning and contrastive learning) consistently underperformed compared to our training-free OT-PCA method. Our analysis indicates that performance inconsistency stems from the LLM's inherent difficulty interpreting representation-based inputs, absent during its pretraining.

Traditional training approaches, despite improving familiarity with molecular representations, fail to guarantee better downstream regression performance. In contrast, our training-free ICRL enables the LLM to interpret FM representations effectively at test time, providing superior performance with minimal computational overhead.

Since the discussion stage is already halfway through, may I know if our rebuttal addresses the concerns? If there are further concerns or questions, we are more than happy to address them. Thanks again for taking the time to review our work and provide insightful comments.

Best regards,

Authors

We sincerely thank Reviewer S7er for the valuable comments. For the concerns and questions, we provide the following responses.

W1: Unclear Trade-off Between Accuracy and Cost

Thank you for the insightful comment.

To clarify the trade-off between performance and cost, we conducted a comparative analysis of ICRL against four recent LLM-based methods for molecular property prediction and found that ICRL achieves comparable or even superior performance to lightweight fine-tuning methods—while requiring no training and only ~2 seconds of CPU computation.

Specifically, we compare ICRL with instruction tuning via Q-LoRA (I-FT) [1], supervised full-parameter pretraining followed by supervised fine-tuning (S-PT + FT) [2], and unsupervised full-parameter pretraining followed by supervised fine-tuning (PT + FT) [3,4]. The comparison spans training paradigms, resource consumption, training duration, and performance on two benchmark datasets. We will incorporate this comparative analysis into the main text in the next revision to make the trade-off more transparent to readers.

Setup:

-- To ensure fairness, we implement ICRL using the LLaMA-3.1-8B-Instruct model—comparable in inference cost to those used in prior work and feasible on a single GPU. Evaluation is based on RMSE. Notably, ICRL achieves even better performance with larger models; e.g., on ESOL, it reaches 0.839 RMSE using LLaMA-3.1-70B.

-- Few-shot examples in ICRL are selected via stratified sampling from the training set, with 20 samples for ESOL and 50 for Lipophilicity, following the same protocol as in our main paper.

-- Regarding computational cost, we extract information directly from the original papers whenever possible. For example, [1] reports using 4× A800-80G GPUs for I-FT, and [3] states that pretraining took 11 days. When GPU specifications or training times are not provided, we estimate them based on similar work. Performance metrics are collected from: [4] Table 3; [3] Table 8; [1] Table 2; and [2] Table 3. Standard deviations are reported only when available in the original texts.

Analysis:

-- Under comparable inference cost, ICRL achieves performance on par with or even superior to that of lightweight fine-tuning and short-duration pretraining + fine-tuning approaches, all while remaining entirely training-free. While lightweight tuning can improve downstream performance to some extent, its effect is limited when the base model has not been pre-trained on small-molecule corpora. In contrast, ICRL allows a general-purpose LLM to achieve competitive or better results with only ~2 seconds of overhead, highlighting both the method's effectiveness and practicality.

-- Improved understanding of small molecules does not necessarily translate to better performance on downstream tasks. In analogy to LLM pretraining, the pretraining in baseline methods is primarily intended to enhance molecular understanding. However, as shown in [3], the standalone pretrained model performs significantly worse than ICRL on downstream prediction, suggesting that pretraining alone mainly serves as a warm-up for subsequent supervised tuning.

Conclusion:

While a performance gap still exists between ICRL and full PT + FT pipelines, ICRL offers comparable or even superior results to supervised lightweight tuning—without any training—underscoring its practical value in low-resource or rapid-deployment scenarios.

---

W2: Method Requires Careful PCA Tuning

Thank you for the comment.

We would like to clarify that the analysis presented in Figure 3(a) and 3(b) focuses specifically on text-level injected representations, aiming to show that more expressive representations do not always help the model better utilize FM features. In contrast, embedding-level injection methods such as OT-PCA are much less sensitive to the PCA dimension—particularly when combined with ICL, as demonstrated in Appendix F.2.2 and Figure 6.

In fact, across all modalities and datasets used in both the main paper and the rebuttal, we consistently set the PCA dimension to 20. This parameter was not tuned during method implementation; the figures simply illustrate a special case analysis rather than a requirement for manual adjustment.

---

Q1: Potential of Lightweight Learnable Projectors

Thank you for your insightful comment.

As shown in W1, while lightweight fine-tuning can improve performance to some extent, it still underperforms compared to the training-free ICRL method—largely due to the inherent difficulty of molecular property prediction tasks for general-purpose LLMs.

To further investigate this question, we explored two lightweight training strategies for the random projectors used in ICRL: one based on contrastive learning (inspired by CLIP), and another based on molecular captioning, following concurrent work. However, as shown in Reviewer J6ma Q2, both approaches perform worse than our OT-PCA method.

We conducted additional experiments by directly training the projector on the downstream task, and found that this approach led to lower predictive performance and impaired generation quality, especially in few-shot settings. Specifically, due to the limited scale of the datasets and the lack of domain-specific knowledge in the LLM, this setup often degraded the model's in-context learning ability—sometimes even preventing it from producing complete outputs.

These findings suggest that for these small molecular datasets, lightweight training of the projector—while feasible in principle—tends to result in worse performance compared to our training-free design.

---

Q2: Scalability of OT Alignment with Model Size

Thank you for the comment.

We would like to clarify that the OT alignment process does not scale significantly with the size of the LLM or the dimensionality of the FM features. The table below summarizes the runtime of OT under different LLaMA model sizes and FM feature dimensions. Importantly, like traditional pretraining or fine-tuning, OT alignment is a one-time computation.

During inference, applying OT requires only a single matrix operation per sample, which takes under 1 millisecond in our setup—adding negligible overhead to the overall generation process. We provide a more detailed discussion of ICRL's inference cost in Appendix E.7 and Table 12.

---

Q3: Limitations of ICRL on Certain Modalities or Tasks

Thank you for your insightful comment.

Indeed, our extensive experiments suggest that while ICRL can enhance LLM performance by leveraging FM representations, the degree of improvement is closely tied to the "difficulty" of the downstream task for the LLM—roughly indicated by its ICL performance.

Specifically, although ICRL outperforms lightweight training approaches on molecular property prediction and molecular QA tasks, it performs less effectively on caption generation tasks (see Reviewer aUho W2). In these cases, the base ICL performance is already poor, suggesting that general-purpose LLMs struggle to learn such complex mappings from a few examples alone. While ICRL still outperforms ICL in this setting, the gap between ICRL and task-specific expert models remains substantial.

A similar limitation appears in protein property prediction tasks. Due to the high sequence similarity within protein datasets, it becomes difficult for the model to distinguish between samples, leading to weak ICL and ICRL performance (see Appendix E.3). This issue, however, can be mitigated by adopting stronger FMs, as discussed in Reviewer J6ma W1.

---

[1] MolecularGPT: Open Large Language Model (LLM) for Few-Shot Molecular Property Prediction. Arxiv 2023.

[2] GIMLET: A Unified Graph-Text Model for Instruction-Based Molecule Zero-Shot Learning. NeurIPS 2023.

[3] SELFormer: Molecular Representation Learning via SELFIES Language Models. Machine Learning: Science and Technology 2023.

[4] GPT-MolBERTa: GPT Molecular Features Language Model for molecular property prediction. Arxiv 2023.

---

As reflected in our title, ICRL serves as a case study in our broader exploration of how LLMs can reason over non-text modalities in a training-free manner. We sincerely appreciate all the valuable feedback provided by the reviewers and will incorporate the insights and suggestions from this rebuttal into the revised version of the paper or pursue them in future work.

Dear Reviewer S7er,

Thanks so much again for the time and effort in our work. Considering the limited time available and to save the reviewer's time, we summarized our responses below.

[W1: Unclear Trade-off Between Accuracy and Cost]

Response: We conducted a comprehensive comparative analysis of ICRL against recent LLM-based molecular prediction methods. Results demonstrate that ICRL achieves performance comparable to or even superior than lightweight fine-tuning methods—with no training required and just ~2 seconds of CPU computation. We will include this detailed analysis in the next revision for transparency.

[W2: Method Requires Careful PCA Tuning]

Response: The PCA dimension parameter (set consistently at 20) was not tuned during implementation. Sensitivity highlighted in Figure 3 pertains specifically to text-level injections. Embedding-level injections, such as OT-PCA used in our method, exhibit robustness to PCA dimension, especially combined with ICL, as detailed in Appendix F.2.2.

[Q1: Potential of Lightweight Learnable Projectors]

Response: Experiments with lightweight learnable projectors (contrastive learning and molecular captioning) consistently underperformed compared to our training-free OT-PCA method, particularly on small molecular datasets. Training projectors directly degraded performance, reinforcing the advantage of our training-free design.

[Q2: Scalability of OT Alignment with Model Size]

Response: Optimal Transport alignment computation time does not scale significantly with LLM size or FM feature dimension. OT is a one-time computation, and inference-time overhead is negligible (under 1 ms per sample). See detailed results in Appendix E.7 and Table 12.

[Q3: Limitations of ICRL on Certain Modalities or Tasks]

Response: ICRL performance improvements correlate with task difficulty for general-purpose LLMs. While outperforming lightweight training approaches on molecular prediction and QA tasks, ICRL shows limited gains in tasks inherently challenging for few-shot learning, such as caption generation and protein prediction with high sequence similarity. Stronger foundational models can mitigate this limitation (discussed further in Reviewer J6ma W1).

Since the discussion stage is already halfway through, may I know if our rebuttal addresses the concerns? If there are further concerns or questions, we are more than happy to address them. Thanks again for taking the time to review our work and provide insightful comments.

Best regards,

Authors

Dear Reviewers and Area Chairs,

Following up on our earlier summary of key discussion points, we provide below a detailed overview of our planned revisions in response to reviewer feedback.

Textual Revisions:

We will revise the manuscript to incorporate the following changes:

• In Section 3, we will add a detailed description of the stratified sampling strategy used for in-context example selection, including the binning-based procedure.

• In Section 4, we will:

  • Clarify the use of multiple regression metrics (RMSE, Pearson's r, and Spearman's ρ), and state that experimental conclusions are consistent across these metrics.
  • Include a discussion on Tanimoto similarity-based selection, and compare its effectiveness with stratified sampling.
  • Add a comparative analysis of training-free and lightweight fine-tuning methods, supported by a cost-performance table across six methods for molecular property prediction.

• In Section 7, we will:

  • Expand the discussion on how representation similarity and embedding variance influence ICRL performance across modalities.
  • Analyze failure cases of lightly-trained projectors (e.g., captioning- and contrastive-based training), and contrast them with the robustness of the training-free approach.
  • Highlight limitations of ICRL on tasks with high inter-sample similarity (e.g., protein datasets), and discuss how stronger foundation models can mitigate these challenges.

---

Experimental Revisions

We will conduct and report the following experiments, with corresponding tables and implementation details added to the appendix, and the main findings discussed in the main text:

• In-context example selection: Evaluate the use of Tanimoto similarity for selecting examples, and compare with stratified sampling in terms of ICL and ICRL performance.

• Cost-performance comparison: Provide a comparative table across six baseline methods, reporting training cost, inference time, and regression accuracy on ESOL and Lipophilicity datasets.

• Protein modality evaluation: Compare different FM encoders (e.g., ESM vs. ProtBert) to assess how embedding similarity influences performance, and to demonstrate that key conclusions—such as the relationship between representation diversity and model effectiveness—are modality-agnostic.

• Task diversity experiments: Report results on molecular QA and caption generation tasks to evaluate ICRL beyond regression settings.

• Lightweight projector training strategies: Evaluate three variants—caption-only, contrastive-only, and combined training—on both generation quality (BLEU/ROUGE) and regression performance (RMSE), and analyze failure modes relative to the training-free baseline.

Again, we thank all reviewers and ACs for your thoughtful comments and support. We are excited to incorporate these revisions and continue refining our work. Should there be any additional questions or suggestions, we would be more than happy to discuss further.

Best regards,

Authors