Foundation Molecular Grammar: Multi-Modal Foundation Models Induce Interpretable Molecular Graph Languages

The authors emphasize interpretability as a major advantage of using FMG. However, the paper only provides limited examples.... A deeper discussion or demonstration (e.g., a domain-expert walkthrough...) would reinforce the claim that this approach is uniquely transparent or insightful.

• The examples in Table 3 are actually drawn from one of the five full example domain-expert walkthroughs in Appendix D, where the expert critiques FMG’s reasoning step-by-step. Additionally, in Appendix D.6, we include concluding thoughts delivered by the expert, highlighting LLM agents’ strong performance in substructure extraction and limits in harder tasks needing expert intuition. The modular nature of the decomposing process is key to FMG’s interpretability, and the case studies highlight how FMG makes this clearer.

Additional Case Study: As both you and V4pu suggested, we ran another case study where an expert contrasted discrepant decompositions by their rationale. Due to the short rebuttal window, we only finished this for 11 chain extenders but are actively working on other datasets. The setup is: the expert chooses a (good, bad) pair of decompositions for each molecule and summarizes the rationale. We then ask our LLM-as-a-judge to summarize the key points and decide solely based on the explanations, which decomp. is better. In most cases, the LLM judge identifies the good decomposition, showing how LLM explanations are useful in closing its own design loop.

For example, the expert wrote:

“Analysis A demonstrates a solid understanding of motifs like amide, urea, imidazole, and carbonyl groups—key to chain extenders. It justifies motif 1 as the root based on its role in peptide bond formation. Analysis B neglects important groups and lacks a strong rationale for selecting motif 0 as root.”

GPT similarly judged:

Defining Motifs:

Analysis A better identifies functional groups like amides/carbonyls that define chain extender properties. It correctly selects motif 1 for its role in polymer backbones.

Functional Group Explanation:

Analysis A explains the contributions of groups to mechanical and processing properties. Analysis B simplifies the motifs and lacks depth in chemical reasoning.

Decision:

Analysis A is favored for better understanding and detailed explanation of groups defining chain extenders.

We repeated the 11-molecule test 5 times and found LLM agreement with the expert on (10, 7, 8, 8, 9) runs.

This case study mirrors expert round-table discussions, where competing rationales must be contrasted to make progress. In App. C, we show selecting higher-ranked decompositions judged by the LLM improves class membership; other design goals could benefit similarly.

We hope this presents further quantitative evidence for the usefulness of the explanations.

While the paper repeatedly highlights the novel idea of using a multi-modal foundation model (MMFM) to induce a Foundation Molecular Grammar (FMG), it is still not evident how this method solves a pressing issue in molecular discovery workflows.... the field of molecular generation typically uses larger, more standard benchmarks to compare performance comprehensively. Without these broader benchmarks or additional state-of-the-art methods, the generality and competitiveness of the proposed approach remain unclear.

• Thanks for these two comments, which go hand-in-hand. We believe the real issue in molecular discovery is the lack of small, domain-specific benchmarks supporting interpretability and expert input. Standard benchmarks focus on large-scale representation learning, and less so on automating interpretable, expert-knowledge-guided design. On the other hand, domain-specific designs in our experience often come with only a handful of training examples, both because the domain is narrow and because the property values of interest are costly to obtain.

• To evaluate FMG’s generality and competitiveness, we do compare with pretrained and transfer learning models in Tables 1 and 2. We see their performance drops significantly due to failing constraints like synthesizability, class membership while retaining broad coverage. When tackling synthetically accessible chain extenders with amine groups, we believe expert knowledge plays a critical role—but integrating it via annotations traditionally required costly manual labor. FMG seeks to automate this labor, doing the hierarchical decompositions while respecting explicit/implicit constraints, while staying interpretable for expert validation.

The notation in tables needs to be well-defined. For example, what does each column mean in the table?

• They are standard metrics used by prior work (e.g. [1,2]). We’ll clarify these notations in the revision.

[1] Data-efficient graph grammar learning for molecular generation. ICLR 2022.

[2] Representing Molecules as Random Walks Over Interpretable Grammars. ICML 2024.

The claims are reasonable and the method itself is well-justified, albeit quite complex, involving many building blocks. I would like to see an ablation study on each building block.

• Thanks for acknowledging the reasonableness of our method! We do have an ablation study on each building block in Section 5.1 (Table 4). In the section, we isolate the importance of the MMFM’s role at each stage of the algorithm. We study the effect of ablating the MMFM with a known heuristic, along with a brief rationale. We also have an ablation of the FMG self-optimization in Sec 5.2 (App. C).

The reported numbers don’t seem to be significant compared to the (much simpler) baselines… presentation of results are problematic: (i) lower scores are often bolded (e.g. in Tab. 1), and (ii) no error-bars are reported.

• Thank you for pointing out these observations. Evaluating generative models is challenging and requires a holistic consideration of different, competing metrics. A method which scores high on one metric (e.g. synthesizability) but does catastrophically bad on another one (e.g. uniqueness) is not practical. When simple fixes don’t work (e.g., tuning sampling temperature), we make a note of it in the caption and exclude it from the rankings.
• Regarding (i), lower scores. In Tab. 1, the two T5 (I) methods are excluded from ranking (as mentioned in the caption) due to the difficulty in obtaining sufficient valid, unique samples, meaning their seemingly higher RS/Memb. scores don’t have sufficient sample support. The same is written in the caption of Tab. 2 for the VAE (T) methods.
• Regarding (ii), no error bars. In this case, robustness from multiple runs (the purpose of error bars) can simply be absorbed into the number of samples we generate. Generating a sufficiently large sample size can ensure greater statistical significance of the results, especially when 4 of our metrics (valid, novel, RS, memb.) are defined at the individual sample level and the other 2 metrics (diversity, uniqueness) evaluate coverage. For small molecules (Tab. 1), we generate 10000 samples; for large molecules (Tab. 2) we generate 1000 samples due to inference being more expensive for some of the large model baselines. In both cases, we see a fixed sample size of 1000 or 10000 already pushes multiple baselines to the limits, so generating more than what they’re capable of may further unbalance the comparisons (see (i)).
• To facilitate a more holistic validation, we made the following spider plots: https://ibb.co/k62nj3c8 https://ibb.co/7JK24Y82
• Only the methods that lie on the Pareto frontier across all metrics are plotted.
  • On small datasets, no method other than FMG reaches near 100% unique, valid & memb. simultaneously. For instance, GPT4 (ICL) appears to score higher on diversity & RS, but struggles to generate valid and unique samples.
  • On real-world datasets, FMG generates the most unique, novel, valid & diverse samples, and methods that score higher on RS and Memb. have serious shortcomings.
• We will add the new plots to the paper to mitigate the challenge of evaluating different generative models by using competing metrics.

...the main point of the method: interpretability, lacks validation. It is unclear to me whether the explanations outputted by the model are actually useful. I would strongly suggest an additional study to quantify this.

• The explanation outputs are critical for the evaluation, interpretation and self-improvement of FMG.
• Evaluation: The GPT-generated explanations are assessed by human experts. We actually have 5 in-depth step-by-step walkthroughs in App. D, with 1 example molecule chosen per dataset (domain). Each step in each case study is scrutinized by an expert (see “Comments by Expert” highlighted in light purple).
• For a quantitative assessment, we tallied the GPT4’s turn-by-turn answers across the 5 case studies by whether the expert completely agrees or not. We had the expert classify the difficulty of the task in each turn.

• There were no instances where the expert thought GPT4’s explanation was flat out wrong. In all but two instances, the expert completely agrees with GPT4’s explanation.
• Interpretation: Additionally, in App. D.6, we include concluding thoughts delivered by the expert, highlighting LLM agents' strong performance in substructure extraction and limits in harder tasks requiring expert intuition.
• Self-improvement: The explanations directly influence the final grammar by summarizing the reasoning taken to achieve the decomposition. They are the inputs for our LLM-as-a-judge protocol, which pits discrepant decompositions against each other (Sec. 3.5) in a tournament, and decides which decompositions to use on the basis of the explanation quality. See our additional case study in our response to PGBN.

The paper uses LLMs to replace heuristics-based rules in molecular grammar methods, which is a novel and interesting way to incorporate domain knowledge into the process of molecular grammar learning.

Thanks for recognizing the novelty, soundness, and intrigue of our work!

Why did the authors use images to describe the molecules to the LLMs? Did the authors experiment with text-based formats like SMILES and SELFIES?

There are three reasons why we choose rendering molecular images as the input to LLMs:

1. Literature review of LLM’s chemistry comprehension abilities: Foundation models like GPT-4o and Gemini have shown multi-modal comprehension ability in aligning natural language descriptions with corresponding images for advanced reasoning, but there has been less work on aligning formal languages like SMILES with corresponding images. This may be because LLMs find it easier to obtain semantics from natural descriptions rather than SMILES. For instance, studies like [1,Section D] and [2] find LLMs perform great on tasks that require reasoning from the natural description of molecules, but struggle when fed SMILES/SELFIES (e.g. 0% SMILES-to-name success in [1] Section D).
2. Technical formulation is in terms of hypergraphs Our underlying formulation builds off the history of hyperedge-replacement grammars, which operate at the hyperedge (or substructure) level instead of the atom level. SMILES/SELFIES syntax doesn’t easily support substructure-level annotations and extending it to do so not only is out of the scope of this work but also changes the data used for LLM pre-training. Meanwhile, rdkit has a library of functions for highlighting, color-coding, and rendering substructures.
3. Interpretability of the grammar learning process. We thought long and hard about how the complex, hierarchical structured representation of hypergraphs can be fed into LLMs. After initial conversations with chemists, we came to the conclusion that highlighting substructures is the most visually meaningful way to consume the information. We, in parallel with a few others [3], identified an opportunity for combining the latest multi-modal understanding and cheminformatics rendering tools. We hope these points sufficiently motivate the visual representation input, and a summarized discussion will be added to the main text.

[1] White et al. Assessment of chemistry knowledge in large language models that generate code. Digital Discovery, 2(2):368–376, 2023.

[2] Guo et al. What Can Large Language Models Do in Chemistry? A Comprehensive Benchmark on Eight Tasks. NeurIPS, 2023.

[3] Wang, Zixu, et al. "Image-based generation for molecule design with SketchMol." Nature Machine Intelligence (2025): 1-12.