FEABench: Evaluating Language Models on Real World Physics Reasoning Ability

Agentic Research on Code Generation vs. LLM-Agent Work

My concerns pertain to the broader domain of LLM-Agent research, not just "agentic research on code generation" claimed in your response, which is only a subset of this domain. You can refer to this paper list for this field, https://github.com/WooooDyy/LLM-Agent-Paper-List.

Concerning the "REAL WORLD PHYSICS REASONING ABILITY" Claim

Your response does not fully address my concern: does your benchmark and evaluation methodology truly assess the "REAL WORLD PHYSICS REASONING ABILITY" claimed in the title? As you mentioned, the problems in your benchmark are solvable using COMSOL alone. However, real-world physics reasoning typically exceeds what a single tool like COMSOL can achieve. It often involves integrating multiple tools, methodologies, and problem-solving techniques, such as combining FEA with analytical methods, probabilistic modeling, or other simulation tools like ANSYS or Abaqus.

On the GUI vs. API/CLI Focus for COMSOL

Your position does not justify entirely disregarding the GUI aspect of COMSOL in your benchmark. 1. The scenarios in SWE-bench and COMSOL differ fundamentally: while programmers often do not depend on a GUI for code-related tasks, engineers using COMSOL rely highly on its GUI for complex workflows. This distinction is critical. 2. Recent advancements in Vision-Language Models (VLMs) show significant progress in interacting with GUIs, as demonstrated by works such as CogAgent[1] and SeeClick[2]. Neglecting the GUI aspect limits the realism of your benchmark, as GUI interactions are integral to the real-world usage of COMSOL. If your rationale for excluding GUI interactions is that "LLMs struggle" with such tasks, it raises concerns about the completeness of your evaluation. By this logic, if LLMs perform poorly on your dataset, it might imply that LLMs should not be used for testing at all, which is a conclusion I don't believe you intend to make.

[1] CogAgent: A Visual Language Model for GUI Agents, https://arxiv.org/abs/2312.08914

[2] SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents, https://arxiv.org/abs/2401.10935

We are grateful for your constructive feedback and have updated our work accordingly.
1. The FEABench Gold ...
We acknowledge the concern about size and hope to address it in future work. FEABench Gold has 15 manually verified problems and 200 algorithmically parsed problems– FEABench Large. However, it can be challenging to scale verified problem generation in this domain. SWE-bench ([1]), when originally released, was an algorithmically parsed dataset with software engineering problems derived from Github issues. SWEBench-Verified, subsequently released by OpenAI earlier this year, consisted of 500 problems verified by a team of human annotators. The OSWorld dataset consisted of 369 manually annotated problems consisting of problems that involve interacting with everyday use apps on Operating Systems.

2. The work is ...

Our primary objective behind the agent was to design the building blocks that would comprise an interactive LLM-Multiphysics Agentic interface, and to demonstrate a possible path toward enhancing the ability of the LLMs to generate code that solves such problems. The landscape of recent agentic research on code generation has largely targeted software development in languages such as Python that are well-represented in training data. SDE workflows often require agility and efficiency in non-trivial file navigation, a task that is not the most relevant skill to the workflow of an FEA engineer. As a concrete example, solving a problem in FEABench Gold does not require a user / LLM to navigate changes to pre-existing code across multiple modules. Even more involved FEA problems would at best involve loading parameter files and geometries files, neither of which resembles SWE workflows.

Revisions: We acknowledge your observation and have better placed our work in the context of recent work devising agentic frameworks for general-purpose Python programming and further added references to related work in Section 5, Discussion and Related Work. We have also rewritten Section 5 (Agent-Multiphysics API Interface) to better clarify our focus on designing the building blocks of possible multiphysics reasoning agents as opposed to introducing an general-purpose agentic algorithm.

3. The experimental evaluation...
We added evaluations of Open Source LLMs and a Fine-Tuned LLM and address this concern in Point 1 of the overall response.

4. The study only ...
We hope we have addressed this concern in Point 2 of the overall response.

5. The paper does ...
We address this comment in Point 3 of the overall response.

We thank you for your time and look forward to addressing any other queries during the discussion phase.

References: [1]: Jimenez et al 2023. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? arXiv:2310.06770
[2] https://openai.com/index/introducing-swe-bench-verified/

We are grateful to you for your comprehensive review and appreciate your recognition of the potential relevance of our work.

Lack of Comparison with Human Performance: In the status quo we manually verified that the answers to the FEABench Gold problems could be computed using the software and that the problem inputs were self-contained while generating / specifying the problem statements. We concur that an extensive comparative benchmarking with humans possessing various degrees of expertise would be of interest. A confounding factor in such a comparison would be the different modalities in which humans / LLMs are likely most adept at interacting with the software, since many users will likely exclusively use the GUI.

Limited Success of LLMs:

• Source of Difficulty – Skills: We introduced the ModelSpecs and Plan versions of the task on FEABench Gold to understand whether giving an explicit natural language (NL) plan enables the LLMs to solve the problem. The Plan2Code version of the tasks probes specifically the ability to translate predefined explicit NL instructions to code. The ModelSpecs2Code task requires the ability to make correct engineering decisions to represent the physics / geometry and translate it into code. A hypothetical LLM that excelled at translating the NL instructions to code but made poor engineering decisions would find the Plan2Code task significantly easier than the ModelSpecs2Code task. The observation that the LLM does not perform significantly better on the Plan2Code task than on the ModelSpecs2Code task indicates that translating NL instructions encoding the decisions to code is also a significant bottleneck.
• Source of Difficulty–Across Problem Types: Figure 4 examined blockwise executability across the different conceptual substeps that comprise each FEA problem. The physics block has the lowest executability upon a single query, which motivates our focus on measuring performance on the physics steps. Figure 6 generates the same plot, contrasting blockwise executability across the “best” states after the agentic interaction with the initial distribution.

Additions: For greater clarity, we provide a detailed qualitative comparison between the LLM’s solution and a ground truth solution in Appendix F and Figure 8.
The ability of alternative architectures or approaches at excelling at this task is an intriguing prospect: we have further discussed our bottlenecks in the Discussion and possible directions of exploration that might mitigate them. Approaches designed to boost the performance of LLMs on low-resource languages (languages without heavy representation in training data) and fine-tune on larger context lengths will likely benefit this domain.

Lack of LLMs Candidates:
We address this in Point 1 of the Overall Response. CodeGemma-7B-IT, in particular, was trained on an additional 500 billion tokens that included mathematics datasets and code repositories ([1]).

Complexity of Task:

• Task Complexity: One uniform way of probing task complexity across problems involves decoupling the ModelSpecs and Plan tasks.
• While it would be interesting to explore other dimensions affecting complexity, given the multistep nature of the problem, other ways of slicing up the problem might not control for other variables that affect difficulty.
• GUI Integration: Discussed in Point 3 of the Overall Response

References: [1]: CodeGemma Team 2024. CodeGemma: Open Code Models Based on Gemma arXiv:2406.11409

W3: I’m not an … the LLM’s understanding.

That is indeed an important question, and one that we have probed through multiple angles.

1. ModelSpecs vs Plan Tasks: We introduced the two versions of the tasks in FEABench Gold to understand precisely this question. The ModelSpecs2Code task requires the ability to both “understand the physics” and make correct engineering decisions to represent it as well as translate it into code. The Plan2Code version of the tasks probes specifically the ability of “understanding how to turn the physics into code” i.e. Plan2Code requires the ability to correctly translate explicit natural language descriptions into code. The observation that the LLMs do not significantly outperform on the Plan2Code tasks than on the ModelSpecs2Code tasks indicates that translating correct decisions to code is also a significant bottleneck.
2. Evaluation: The multipronged evaluation scheme also serves to decouple different aspects of how close the LLM is to solving the problem. For example, a model with 100% executable code might not be generating code that is relevant to solving the problem. Thus the Model Tree Score measures the alignment of the LLM solution path (tree) with that of a ground truth solution path.
3. Boilerplate code: We show a qualitative example of the code generated by an LLM and a ground truth solution in Appendix E. The 3 SOTA LLMs we tested are able to adhere to the overall “boilerplate code” structure in the ground truth solution, and it is the options / arguments they pass to these calls that differ. The physics metrics also evaluate the options within the boilerplate calls and this example analyzes how the metrics flag inconsistencies for this example.

Cost and Runtime:

The agent experiment took around $2 per problem (as an approximate upper bound). Our agent experiments take around slightly over 12 minutes (between ~7-17 minutes) on average per problem. The FEA runtime is only a minuscule fraction of this time: parsing the LLM reply, evaluating it by executing it in COMSOL Multiphysics® and retrieving API messages took around 0.9-1.5s for a single LLM reply. We added these statistics to Appendix E.

We hope we have addressed your questions in our revision and respectfully hope you might consider re-evaluating our work.

References: [1]: Jimenez et al 2023. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? arXiv:2310.06770

We are grateful for your thoughtful review of our paper.
W0: Having to get …to describe problems.

1. Addressed in Point 2 of the Overall Response
2. Although several universities and engineering organizations have academic licenses for their students / staff to use COMSOL Multiphysics ®, we acknowledge and are cognizant of barriers that can impede accessibility. Unfortunately, many commercial-grade softwares require paid licenses. Eschewing widely-used FEA softwares would not provide the LLMs with the most authentic mirror of the environments FEA engineers use.
3. We appreciate your thoughts on this concern, and ran a baseline attempt to query the LLMs to solve the FEABench Gold problems by generating Python code using Python numerical packages and FEniCS. We used the * same * problem description as for the COMSOL Multiphysics ® experiments. The metrics we can use for this baseline are only whether a single sample of the code (without iterative interaction) was executable and the target relative error if a value was computed. Only one generated code was executable, although the answer it computed had a greater than 100% relative error with respect to the correct answer.

W1: It feels like … I thought it said 1 shot.

In terms of adding increasing amounts of documentation, we ran 3 versions of experiments on FEABench Gold for task ModelSpecs:

• One-Shot (LLM sees a single example of problem -> code in-context)
• One-Shot + Physics Documentation In-Context (the LLM sees the list of valid physics API options for interfaces and features * in addition to * the One-Shot example in-context)
• Multi-Turn Agent: Here we first curated an LLM-annotated corpus of (NL Annotation -> Code Snippet) pairs, decomposed by the block of code. One of the tools in the agent enables the LLM to phrase a natural language query for a directed step (eg: “define an axisymmetric geometry”), and the retriever returns the 3 closest examples of (NL, Code) pairs. This is added to the context of the CorrectorAgent in the Agent experiment that can then use these examples of correctly formatted code to return the next solution.

Additions:

1. We have restructured the presentation in Section 5 to make the last of these components more clear.
2. Does fine-tuning boost performance: Added in Point 4 of the overall response and Appendix G.

W2: The benchmark … to some extent.

While a completely unsolvable benchmark may seem harder to track incremental progress on, we introduced the multifaceted evaluation scheme precisely as a way to assign ‘partial credit’ to solutions and glean signals of improvement. We find that our baseline agent is able to increase its ability to generate executable code over the course of a single experiment of 15 minutes.

Moreover, considering the phenomenal pace of developments in LLM and agentic research, we contend that the (current) difficulty is an advantage, since an easier benchmark is more likely to be saturated soon after it is released. SWE-bench ([1]), one of the leading benchmarks spurring LLM development today, only had 1.96% of problems solvable when it was released a year ago.

The iterative nature … for simulation.

In our agentic experiments, parsing the LLM reply, evaluating it by executing it in COMSOL Multiphysics® and retrieving API messages took around 0.9-1.5s for a single LLM reply. Our agent experiments take around slightly over 12 minutes (ranging from 7-17 minutes) on average per problem. Several challenging benchmarks, even in high-resource languages such as Python, have required iterative interaction to solve. Other work ([1]) that has focused on data science problems for example, the best performing agent is given 24 hours to solve a single problem. Nonetheless, we added a Fine-Tuning experiment (Point 4 of the overall response), since that is likely a plausible route to enhance solvability without multiple iterations.

I would appreciate … in this work. We hope we have addressed this in the changes made in the revision (changes made in response to your feedback). The unique elements we adopted for our setting include:

• Analytical-Numerical Consistency: If the VerifierLLM component of the Evaluator is asked to critique the solution, it compares the LLM’s computed numerical answer with its analytical guess.
• LLM-Assisted Semantic Code Search: Since our experiments demonstrate that at least part of the challenge is from the lack of familiarity of the LLM with the code snippets. We used an auxiliary LLM to annotate individual compositional blocks that comprise an entire code. While the specific corpus we use is COMSOL-specific, the approach is not, and can be generalized to other low-resource programming languages (LRPLs) or other domain-specific coding languages.

We are grateful to you for your insights and feedback and have modified our draft accordingly.

References: [1] Chan et al 2024. MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering

We thank you for your valuable feedback and appreciate your recognition of the utility of our evaluation metrics.
Changes made in response to your feedback:

• Comparative Analysis and Agent Design: We reframed Section 5, 7 and the Related Work Appendix, and strengthened the ways in which we are related to existing agentic approaches in the literature as well as distinguish ourselves from them in Related Work (discussion on Reflexion), and the Discussion.
• Structure: Separated the elements related to the Multiphysics-Software Interface from the rest of the agent control flow
• Feedback: Specific point under Design Elements in Section 5
• Task Handling: Discussed in Section 5.1. (To minimize…input context)
• Unique aspects of our design: Described in Design Elements.

Response:

Limited Success:

• Our objective behind introducing the agent was to devise a best-effort baseline that seeks to address obvious sources of difficulty in the single-turn setup, such as the absence of interaction with and knowledge about the API. We find that in spite of curating such an environment, the problems remain a challenge. Our objective was not to show that our existing agentic setup is capable of completely solving the benchmark, but rather to design an LLM-Agentic interface for this domain that finds a path toward mitigating some of the issues in the single-turn setting, as is observed in the improved executability-focused metrics.
  Moreover, if the agent was, in fact, able to perfectly solve these problems, this would indicate that the benchmark would be easier to solve, and would thus pose less of an “unsolved” challenge to the community.
  Changes: A side note about the lack of improvement in “code similarity” in particular: As indicated in the evaluation section (L174-175 in the submitted draft), we mainly chose to include code similarity score as a baseline metric for code similarity. The equivalence between multiple blocks of code and the large amount of boilerplate code dilutes the use of this metric and it has very little variation across all experiments, and not just the agent. We have clarified our description of Code Similarity in the revised version to address this. L181-184: “We mainly report this metric as a baseline measure of code similarity, and to motivate our introduction of domain-specific metrics. The preponderance of boilerplate syntax, along with the fact that two different code blocks could generate equivalent model subtrees, are factors that contribute to the lack of meaningful variation of this metrics across experiments.”

Absence of Agent Baselines:

To the best of our understanding, SOTA agent frameworks cannot directly be used off-the-shelf in an optimum fashion for our software setting. Thus a direct quantitative analysis would not be an apples-to-apples comparison. The recent landscape of agent research has focused on code generation in general-purpose languages such as Python which often requires mastery over skills like localizing and fixing bugs across multiple files. For example, the key components frameworks like [1], [2], [3] that excel at SWE-Bench ([4]) enhance utilities for file manipulation and leveraging tools from existing Python packages. These utilities are optimized for software engineering workflows but are less relevant to FEA workflows. As a concrete example, solving an FEA problem in FEABench Gold does not require the user / LLM to read in files or navigate changes to pre-existing code across multiple modules or directories.

Where relevant, however, there are conceptual elements that we design for our Agent-Software interface that are shared with other general-purpose agents. Other works ([1]) have used the ability to make specialized tool calls . In our interface, the ToolLookupAgent serves this purpose, as it uses the Langfun interface to call different specialized Python functions that are relevant to interacting with the COMSOL Multiphysics(R) API, such as QueryModelTreeProperties.

Previous work on building agent-software interfaces has underscored the need to carefully design interfaces that are well-adapted to the problem domain and actions that provide feedback in a concise yet information-dense fashion and are cognizant of LLM limitations. We sought to adhere to these principles in designing our agent-software interface tailored to enabling an LLM to interact with an FEA software.

References: [1]: Wang et al 2024. OpenHands: An Open Platform for AI Software Developers as Generalist Agents. arXiv: 2407.16741
[2]: Wang et al 2024. Executable Code Actions Elicit Better LLM Agents. arXiv: 2402.01030
[3]: Yang et al 2024. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. arXiv: 2405.15793
[4]: Jimenez et al 2023. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? arXiv:2310.06770

4. Fine-Tuning on FEABench Large: We added an experiment in which we fine-tuned Gemini 1.5 Flash on a subset of FEABench Large using a zero-shot prompt, to adhere to context limits. We then examined the performance of the fine-tuned checkpoint on FEABench Gold and contrasted it with the performance of the untuned LLM and the untuned LLM with a One-Shot prompt. The fine-tuned LLM does not offer an advantage over the One-Shot prompting scenario. This is likely in large part due to token window limits during fine-tuning. We discuss this in more detail in Appendix G.

We have discussed your other comments in our responses below. We reiterate our gratitude for helping us improve our work.