F-Adapter: Frequency-Adaptive Parameter-Efficient Fine-Tuning in Scientific Machine Learning

Thanks for the thoughtful comments!

Reply to W1 & Q3

First, we evaluated a simpler test case for the base model: the 2D shallow water equations (SWE-2D) from PDEBench. This choice removes any bias from base model limitations when assessing PEFT methods. The results show that F Adapter achieves the lowest relative error (≈1.6%).

Second, we carried out visual comparisons of predicted and ground truth velocity fields using slice plots and isosurfaces. Unfortunately, rebuttal policy prevents us from including these figures here. In broad terms, the F‑Adapter preserves the filament‑like vortical structures and reproduces both high‑ and low‑energy regions, with only small local deviations in amplitude. LoRA, in contrast, yields coarse block‑shaped patches with muted intensities and loses most fine‑scale features, suggesting a collapse toward a spatially averaged flow. These qualitative visual observations support our quantitative error analysis and show that the adapter retains considerably more spectral detail than LoRA in this challenging regime. We will present the full set of visual results in the camera ready version.

Reply to W2 & W4

Firstly, our original manuscript actually evaluated on three very challenging datasets (see Appendix C.5).

Besides the SWE-2D introduced in our Reply to W1 & Q3, we now add another experiment on the 3D magnetohydrodynamic equations (MHD-3D) under extreme data scarcity. We adopt the GL-FNO[1] data processing protocol but train on only 24 trajectories.

The results show that even with such limited data, our F Adapter continues to achieve substantially higher accuracy than the alternative PEFT schemes which fail to adapt under these conditions.

[1] Du, Yutao, et al. "Global-local Fourier Neural Operator for Accelerating Coronal Magnetic Field Model." 2024

Reply to W3 & Q2

Firstly, FFTs (FNO) are still the cornerstone of most LOMs, much as Multi‑Head Attention is for Large Language Models. With elaborate design, it can still stably scale up and unleash powerful pre-trained capabilities. Representative FFT‑based LOMs include

1. PreLowD (TMLR 2024), which relies on a factorized Fourier Neural Operator whose core loop is FFT, inverse FFT, and spectral convolution.
2. UPS (TMLR 2024), whose first stage applies FNO spectral blocks to map the physical field into a token sequence processed by a language‑model body.
3. CoDA‑NO (NeurIPS 2024), where on a uniform grid the key operators K, Q, V, M, and the integral operator I are all implemented with FNO components.
4. OmniArch (ICML 2025), which uses a Fourier encoder to move spatial field values to the frequency domain.
5. DPOT (ICML 2024), our main backbone, which is the only open‑source model in this family with more than 1B parameters and outperforms purely transformer‑based baselines such as MPP in their original paper.

Secondly, although an FNO backbone provides direct access to frequency features, our main focus is on assigning each frequency band its own proper bottleneck dimension rather than strictly performing convolution in the frequency domain. This insight allows our F‑Adapter to extend naturally to non‑FFT architectures. On the pure transformer‑based Poseidon model, we estimate frequency energy for each Linear layer from adjacent‑token differences, and for each Conv2d layer we perform a local real 2‑D FFT on the convolution output to obtain an energy spectrum that guides the adapter’s weight generation. The adapter itself still operates in the native spatial domain. Capacities are allocated to bands according to their energy, following Eq(6) of our paper, which equips the model with frequency awareness. The accompanying table reports the resulting performance gains on 2D Shallow Water Equation Dataset.

We observe that adapters deliver strong results when the base model is an FNO, yet their effectiveness declines sharply on a transformer backbone. In contrast, LoRA and its variants demonstrate robust performance on transformer backbones, reflecting established best practices in fine-tuning LLMs. But our F‑Adapter still narrows this gap by significantly improving adapter performance on transformers. Building on this insight, we introduce F‑LoRA: it preserves the frequency‑based capacity allocation of F‑Adapter while replacing the bottleneck MLP with LoRA‑style low‑rank linear updates. F‑LoRA achieves SOTA performance across a broad suite of PEFT methods in this setting.

Reply to W5 & Q1

Below we present additional spectral results on the 3‑D Turbulence dataset. To quantify how well individual scales are reproduced, we report two metrics:

1. Root mean square logarithmic error (RMSLE) of the energy spectrum

We compare the predicted and DNS energy spectra shell by shell in wave number space. This metric measures their discrepancy on a logarithmic scale and ensures that both low and high wave number bands receive equal weight.

2. Relative error of the total kinetic energy

Here $E_{\text{tot}}$ is the area under the spectrum. Conservation of total energy confirms that this quantity is directly proportional to the volume‑averaged turbulent kinetic energy.

In addition, the camera‑ready version will include a plot of $E(k)$ and the DNS spectrum on logarithmic axes. The experimental results demonstrate that our F‑adapter significantly outperforms both LoRA and the strongest competing baseline, the Vanilla Adapter, in accurately capturing the energy spectrum.

Thanks for the thoughtful comments!

Reply to W1

Thanks for your concerns. We would like to discuss our exploration on non-spectral backbones here.

Firstly, FFTs (FNO) are still the cornerstone of most LOMs, much as Multi‑Head Attention is for Large Language Models. Representative FFT‑based LOMs include

1. PreLowD (TMLR 2024), which relies on a factorized Fourier Neural Operator whose core loop is FFT, inverse FFT, and spectral convolution.
2. UPS (TMLR 2024), whose first stage applies FNO spectral blocks to map the physical field into a token sequence processed by a language‑model body.
3. CoDA‑NO (NeurIPS 2024), where on a uniform grid the key operators K, Q, V, M, and the integral operator I are all implemented with FNO components.
4. OmniArch (ICML 2025), which uses a Fourier encoder to move spatial field values to the frequency domain.
5. DPOT (ICML 2024), our main backbone, which is the only open‑source model in this family with more than 1B parameters and outperforms purely transformer‑based baselines such as MPP in their original paper.

Secondly, although an FNO backbone provides direct access to frequency features, our main focus is on assigning each frequency band its own proper bottleneck dimension rather than strictly performing convolution in the frequency domain. This insight allows our F‑Adapter to extend naturally to non‑FFT architectures. On the pure transformer‑based Poseidon model, we estimate frequency energy for each Linear layer from adjacent‑token differences, and for each Conv2d layer we perform a local real 2‑D FFT on the convolution output to obtain an energy spectrum that guides the adapter’s weight generation. The adapter itself still operates in the native spatial domain. Capacities are allocated to bands according to their energy, following Eq(6) of our paper, which equips the model with frequency awareness. The accompanying table reports the resulting performance gains on 2D Shallow Water Equation Dataset.

We observe that adapters deliver strong results when the base model is an FNO, yet their effectiveness declines sharply on a transformer backbone. In contrast, LoRA and its variants demonstrate robust performance on transformer backbones, reflecting established best practices in fine-tuning LLMs. But our F‑Adapter still narrows this gap by significantly improving adapter performance on transformers. Building on this insight, we introduce F‑LoRA: it preserves the frequency‑based capacity allocation of F‑Adapter while replacing the bottleneck MLP with LoRA‑style low‑rank linear updates. F‑LoRA achieves SOTA performance across a broad suite of PEFT methods in this setting.

Reply to W2

We conducted extensive ablation studies on diverse hyperparameter settings using DPOT and F-Adapter on 3D-Turbulance dataset.

The first row details the default hyperparameter configuration applied across all experiments in our original manuscript. Results indicate that hyperparameters primarily influence performance by modulating adapter capacity allocation across frequency bands. This adjustment effectively governs the model's overall capacity. Crucially, hyperparameters do not directly affect performance. Their impact is mediated through capacity allocation. Consequently, selecting appropriate capacity based on computational resource constraints enables predictable performance outcomes. The magnitude of this impact remains relatively limited. This finding aligns with the insight presented in Table 1 of our original paper, which shows that adapters nearly obey the scaling law. Our Band-Specific Bottleneck Allocation framework (Eq 6) maintains robust generalization across diverse tasks, while hyperparameters retain flexibility to accommodate available computational resources.

Reply to W3

In Table 2, we present a comprehensive comparison between full fine‑tuning, our F Adapter, and 8 alternative PEFT schemes. This evaluation includes methods such as LoRA, Adapter, and Prompt Tuning. Across most benchmarks, all eight approaches perform better than BitFit. Since our main manuscript focuses on frequency‑domain PEFT designs and BitFit only updates bias parameters without delivering targeted improvements or new insights, we did not include BitFit as a baseline.

Reply to W4

We appreciate this perspective on methodological novelty. While our framework integrates established concepts like adapters and spectral decomposition, its core contribution lies in the first unified theoretical and empirical framework comprehensively addressing domain-specific fine-tuning challenges in SciML. Critically, our frequency-aware parameter allocation principle (Eq. 6) extends beyond Fourier-based architectures. When applied to transformer based models like Poseidon through F-LoRA, it still achieves SOTA results, demonstrating broader applicability beyond spectral architectures. This evidence confirms that our high-level methodology generalizes to diverse architectures and tasks. The details have been shown in Reply to W1. These findings reveal a transferable design principle: task performance in SciML is closely linked to the efficient division of labor across features of varying frequencies, regardless of the base architecture. While instantiated here for PDE-solving, this paradigm provides actionable insights for other tasks with strong frequency dependencies.

Reply to Q1

Firstly, please refer to Reply to W1 for the performance of F-Adapter on operator networks without explicit FFTs. Secondly, we supplement our study with a new set of comparative tests on the 2D shallow‑water equations (SWE) and the 3D magnetohydrodynamic equations (MHD). In the SWE setting, phenomena such as hydraulic jumps and strongly non‑uniform interfaces produce shocklike discontinuities and pronounced high‑wavenumber content. The MHD represents a class of PDEs characterized by broadband spectral behavior. Moreover, we only extracted 24 trajectories for HHD training, aiming to evaluate the performance under extreme data scarcity conditions.

Our results demonstrate that F‑Adapter still achieves SOTA. Intuitively, this stems from our frequency‑adaptive allocation of model capacity. Rather than neglecting high‑frequency features, we partition the spectrum so that all frequencies are processed more efficiently and effectively. F‑Adapter combines three complementary mechanisms: wide channels for low‑frequency components, lightweight residual connections for high‑frequency components, and a nonlinear universal approximator to concentrate limited parameters on the most critical features. High‑frequency information, although important, is energetically sparse and can be captured with small bottleneck modules. These modules both preserve essential details and impose an implicit regularization. Consequently, F‑Adapter excels in turbulent flows and other scenarios dominated by high‑frequency dynamics.

Reply to Q2

We would like to clarify that the hyperparameters that determine the model’s dimensionality may be difficult to make adaptive or learnable, since they fix the shape of weight tensors and cannot be updated during training. Nonetheless, we conducted extensive ablation studies on these settings in our Reply to W2.

Reply to Q3

Table 2 in our manuscript includes the accuracy results of full parameter fine tuning for reference, and in several cases F-Adapter attains higher performance.

Reply to Q4

This is a valuable suggestion that aligns with our interest in refining PEFT strategies for SciML. Notably, we have already conducted preliminary experiments on frequency-adaptive LoRA specifically tailored for Transformer-based large operator models, with details available in Reply to W1. We believe these frequency-adaptive LoRA components could potentially be integrated with our current framework to leverage the strengths of both linear and non-linear adaptations. However, such integration requires careful design and extensive validation to ensure compatibility and performance gains, which would demand additional time beyond the current scope. Thus, we plan to pursue this integration as a dedicated direction in our future research.

Thanks for the thoughtful comments!

Reply to W1

Firstly, FFTs (FNO) are still the cornerstone of most LOMs, much as Multi‑Head Attention is for Large Language Models. Representative FFT‑based LOMs include

1. PreLowD (TMLR 2024), which relies on a factorized Fourier Neural Operator whose core loop is FFT, inverse FFT, and spectral convolution.
2. UPS (TMLR 2024), whose first stage applies FNO spectral blocks to map the physical field into a token sequence processed by a language‑model body.
3. CoDA‑NO (NeurIPS 2024), where on a uniform grid the key operators K, Q, V, M, and the integral operator I are all implemented with FNO components.
4. OmniArch (ICML 2025), which uses a Fourier encoder to move spatial field values to the frequency domain.
5. DPOT (ICML 2024), our main backbone, which is the only open‑source model in this family with more than 1B parameters and outperforms purely transformer‑based baselines such as MPP in their original paper.

Secondly, although an FNO backbone provides direct access to frequency features, our main focus is on assigning each frequency band its own proper bottleneck dimension rather than strictly performing convolution in the frequency domain. This insight allows our F‑Adapter to extend naturally to non‑FFT architectures. On the pure transformer‑based Poseidon model, we estimate frequency energy for each Linear layer from adjacent‑token differences, and for each Conv2d layer we perform a local real 2‑D FFT on the convolution output to obtain an energy spectrum that guides the adapter’s weight generation. The adapter itself still operates in the native spatial domain. Capacities are allocated to bands according to their energy, following Eq(6) of our paper, which equips the model with frequency awareness. The accompanying table reports the resulting performance gains on 2D Shallow Water Equation Dataset.

We observe that adapters deliver strong results when the base model is an FNO, yet their effectiveness declines sharply on a transformer backbone. In contrast, LoRA and its variants demonstrate robust performance on transformer backbones, reflecting established best practices in fine-tuning LLMs. But our F‑Adapter still narrows this gap by significantly improving adapter performance on transformers. Building on this insight, we introduce F‑LoRA: it preserves the frequency‑based capacity allocation of F‑Adapter while replacing the bottleneck MLP with LoRA‑style low‑rank linear updates. F‑LoRA achieves SOTA performance across a broad suite of PEFT methods in this setting.

Reply to W2

You raise a valid point. In our paper, we define a Large Operator Model (LOM) as a pre‑trained foundational model that reaches the parameter scale commonly associated with “large models” and learns solution operators for PDEs. The ambiguity you noted comes from our wording: we meant only those LOMs built on the FNO backbone, not models using transformer‑based architectures. We will revise this sentence in the camera ready version to make it clear that “large‑scale operator models” in that sentence refers exclusively to FNO‑based LOMs.

Reply to Q1

While LoRA is not designed to improve performance, it serves as a viable option when memory and computational budgets are limited. Therefore, the goal of this work is to find a better parameter-efficient fine-tuning method that can achieve stronger performance under the same memory and computational constraints, rather than seeking improvements over the full-parameter fine-tuning method. Please refer to Table 9 in the appendix of our paper. When compared to LoRA, F‑Adapter achieves comparable FLOPs and per‑step inference time, while delivering superior performance.

Reply to Q2

SFT adapts a pre‑trained large model to a downstream task by updating all of its parameters so as to minimize the supervised loss on labeled data. PEFT pursues the same objective using labeled data but only introduces or updates a small set of extra parameters, while keeping the backbone weights fixed. In other words, SFT adjusts every model weight, whereas PEFT makes only minimal changes, making it a more resource friendly variant of supervised fine tuning. Both methods share the goal of maximizing task performance; their sole distinction lies in the fraction of parameters that are trainable. We acknowledge the relevance of Chen et al. (2024) [1], which explores data-efficient operator learning via unsupervised pretraining and in-context learning (ICL) in SciML. While their work focuses on reducing labeled data dependency through ICL, a non-fine-tuning approach, our study prioritizes PEFT for scenarios where task-specific labeled data exists but full parameter updates are computationally prohibitive. As shown in Table 2 and Table 9, we have empirically compared the performance and computational cost of full fine tuning against several PEFT techniques.

I would like to thank the author for the detailed responses that addresses my concerns. I will maintain my score as 4.

Thanks for the thoughtful comments!

Reply to W1 & Q1

Firstly, FFTs are still the cornerstone of most LOMs, much as Multi‑Head Attention is for Large Language Models. Representative FFT‑based LOMs include

1. PreLowD (TMLR 2024), which relies on a factorized Fourier Neural Operator whose core loop is FFT, inverse FFT, and spectral convolution.
2. UPS (TMLR 2024), whose first stage applies FNO spectral blocks to map the physical field into a token sequence processed by a language‑model body.
3. CoDA‑NO (NeurIPS 2024), where on a uniform grid the key operators K, Q, V, M, and the integral operator I are all implemented with FNO components.
4. OmniArch (ICML 2025), which uses a Fourier encoder to move spatial field values to the frequency domain.
5. DPOT (ICML 2024), our main backbone, which is the only open‑source model in this family with more than 1B parameters and outperforms purely transformer‑based baselines such as MPP in their original paper.

Secondly, although an FNO backbone provides direct access to frequency features, our main focus is on assigning each frequency band its own proper bottleneck dimension rather than strictly performing convolution in the frequency domain. This insight allows our F‑Adapter to extend naturally to non‑FFT architectures. On the pure transformer‑based Poseidon model, we estimate frequency energy for each Linear layer from adjacent‑token differences, and for each Conv2d layer we perform a local real 2‑D FFT on the convolution output to obtain an energy spectrum that guides the adapter’s weight generation. The PEFT adapter itself still operates in the native spatial domain. Capacities are allocated to bands according to their energy, following Eq(6) of our paper, which equips the model with frequency awareness. The accompanying table reports the resulting performance gains on 2D Shallow Water Equation Dataset.

We observe that adapters deliver strong results when the base model is an FNO, yet their effectiveness declines sharply on a transformer backbone. In contrast, LoRA and its variants demonstrate robust performance on transformer backbones, reflecting established best practices in fine-tuning LLMs. But our F‑Adapter still narrows this gap by significantly improving adapter performance on transformers. Building on this insight, we introduce F‑LoRA: it preserves the frequency‑based capacity allocation of F‑Adapter while replacing the bottleneck MLP with LoRA‑style low‑rank linear updates. F‑LoRA achieves SOTA performance across a broad suite of PEFT methods in this setting.

Reply to W2 & Q2

We would like to clarify that the hyperparameters that determine the model’s dimensionality may be difficult to make adaptive or learnable, since they fix the shape of weight tensors and cannot be updated during training. Nonetheless, we conducted extensive ablation studies on diverse hyperparameter settings using DPOT and F-Adapter on the 3D-Turbulance dataset.

Results indicate that hyperparameters primarily influence performance by modulating adapter capacity allocation across frequency bands. This adjustment effectively governs the model's overall capacity. Crucially, hyperparameters do not directly affect performance. Their impact is mediated through capacity allocation. Consequently, selecting appropriate capacity based on computational resource constraints enables predictable performance outcomes. The magnitude of this impact remains relatively limited. This finding aligns with the insight presented in Table 1 of our original paper, which shows that adapters nearly obey the scaling law. Our Band-Specific Bottleneck Allocation framework (Eq 6) maintains robust generalization across diverse tasks, while hyperparameters retain flexibility to accommodate available computational resources.

Reply to Q3

Firstly, we have reported turbulence experiments in both Table 2 and Table 3. Now we supplement our study with a new set of comparative tests on the 2D shallow‑water equations (SWE-2D) and the 3D magnetohydrodynamic (MHD-3D) equations. In the SWE setting, phenomena such as hydraulic jumps and sharply non‑uniform interfaces generate shock‑like discontinuities and rich high‑wavenumber content, requiring high‑frequency fidelity for accurate capture. The MHD equations represent a class of PDEs that are characterized by broadband spectral behavior. Moreover, we only extracted 24 trajectories for training, aiming to evaluate the performance under extreme data scarcity conditions.

Our results demonstrate that F‑Adapter still achieves SOTA. Intuitively, this stems from our frequency‑adaptive allocation of model capacity. Rather than neglecting high‑frequency features, we partition the spectrum so that all frequencies are processed more efficiently and effectively. F‑Adapter combines three complementary mechanisms: wide channels for low‑frequency components, lightweight residual connections for high‑frequency components, and a nonlinear universal approximator to concentrate limited parameters on the most critical features. High‑frequency information, although important, is energetically sparse and can be captured with small bottleneck modules. These modules both preserve essential details and impose an implicit regularization. Consequently, F‑Adapter excels in turbulent flows and other scenarios dominated by high‑frequency dynamics.