Neat: Nonlinear Parameter-efficient Adaptation of Pre-trained Models

In neural networks, non-linearities are applied to activations, not the weight matrices themselves. Therefore, there is basically no motivation to add non-linearity to model the weight changes.

Thank you for your feedback. We would like to clarify that the non-linearity is present in the model update. Equations (3) and (4) represent iterative model updates. The gradient of $L$ with respect to the model weight is a complex function of $W_0^{0}$, and the goal of PEFT is to approximate the weight update $\Delta W$. Although the transformer layers are nonlinear functions, they can not be used to capture the nonlinearity of the model update because they are frozen during fine-tuning. For the proposed NEAT, since $W_0^{0}$ is the input to this function, we aim to leverage a nonlinear network that takes $W_0^{0}$ as input to approximate the update directly. This approach allows us to efficiently capture more complex and richer transformations of the weights.

Introducing non-linearity into the low-rank matrices A and B may contradict the core idea of "efficient fine-tuning" by adding unnecessary complexity.

In our simplest and most commonly applied settings, we use only the input and output layers with a single activation function (ReLU), without involving intermediate layers. This setup introduces minimal overhead to the runtime and memory usage of NEAT and does not impair its efficient fine-tuning capability. To support our claims, we report the runtime and memory consumption of NEAT and LoRA in Table 1 for the MRPC task. We ran 20 epochs with a parameter count of 0.3M, using other hyperparameters as specified in the paper. The results indicate that NEAT performs comparably to LoRA in terms of runtime and memory consumption (nearly identical), while achieving notably better accuracy. Additionally, we emphasize that NEAT introduces little additional complexity compared to LoRA in its basic form, which is the configuration primarily used in our experiments.

Table 1: parameter efficiency in MRPC

include comparisons of time or memory efficiency.

Thank you for pointing this out! In addition to the results shown in Table 1, we provide further efficiency analysis in Tables 2 and 3, using the COLA and STS-B datasets from the GLUE benchmark. We will include this analysis in the revised version of the paper. It can generally be observed that NEAT performs comparably to LoRA in terms of parameter efficiency. In most cases, NEAT consumes almost the same amount of memory as LoRA and has similar runtime performance. These observations demonstrate that NEAT is an efficient PEFT method with high parameter efficiency. Please note that due to different hyperparameter configurations across tasks, memory consumption may vary.

Table 2: parameter efficiency in COLA

Table 3: parameter efficiency in STS-B

The paper relies heavily on results from other studies, and should reproduce them.

This point certainly warrants further explanation. We conducted several pilot experiments using the open-source codebase provided by the authors and confirmed the reproducibility of their results before directly citing them. Additionally, we observed that some works are not open-source, and to ensure a fair comparison, we report their results as presented in their papers. To improve reproducibility, we will release our code in the final version.

1. Using the authors' formulation in Equation (2), we have $A = \mathrm{optimize}(W^0) = \mathrm{argmin} \mathcal{L}(\mathcal{D}_{\mathrm{train}}; W^0 + AB)$, which makes $A$ non-linear with respect to $W^0$.

2. Your response is too high-level. Please consider me as someone without any prior knowledge of ML memory profiling. Could you provide a detailed memory profile for Neat? For example, specify the actual memory usage for storing pretrained model weights, activations, and gradients.

For Proposition 5.1, there is an assumption that the loss function is invariant under projection to a left singular space. Listing some losses that are invariant under this projection would strengthen that result.

Thank you for the suggestion. We will add more details of the invariance assumption. Here is the intuition behind the assumption. The invariance assumption in Proposition 5.1 pertains to the pre-trained model rather than the loss function itself. Specifically, the assumption posits that the loss, as a function of the parameter weights being fine-tuned, remains invariant under the projection $U U^\top$ onto the subspace spanned by the left singular vectors of the pre-trained weight matrix $W^0$. To illustrate this concept, consider a pre-trained model represented as a composition of three functions: $\mathcal{M}(x) = \mathcal{M}_3 \circ \mathcal{M}_2 \circ \mathcal{M}_1(x)$, where $\mathcal{M}_2(x) = W^0 x$ is the component being fine-tuned. The invariance assumption holds if the downstream function $\mathcal{M}_3$ satisfies the following property: it produces outputs of zero for any input that lies in the space orthogonal to the feature space of $\mathcal{M}_2$ (i.e., the subspace spanned by the left singular vectors of $W^0$). At a high level, this implies that $\mathcal{M}_3$ is specifically designed or trained to only "activate" in response to features produced by $\mathcal{M}_2$ during pre-training tasks. This assumption is reflective of how pre-trained models often encode meaningful structure within their dominant subspaces, while disregarding irrelevant directions. Furthermore, it is worth noting that this invariance assumption becomes unnecessary in certain cases, such as when sinusoidal activation functions are employed. In such settings, as shown in Proposition A.1 (Appendix), the output of the model inherently respects the structure of the feature space, rendering the assumption superfluous.

For Figure 2 (and possibly other experiments), a single run for each depth doesn't provide strong confidence that this trend wasn't from random chance.

Thanks for your valuable suggestion. We run multiple runs for Figure 2 and report the results below.

Table 1: multiple runs for multi-layer NEAT

The trend generally holds that introducing greater depth (i.e., intermediate layers) into the NEAT structure yields better results compared to vanilla NEAT.

The experiments run are only for Neat. The other methods' accuracies come from other work rather than being reproduced, which may result in some differences simply from the training implementation.

Thank you for your comments. First, we use the exact same settings as the compared baselines, ensuring a fair comparison between our results and the baseline results. Additionally, some baselines did not release their code. To ensure a fair and convincing comparison, we directly report the results from their papers, which is a common practice for comparing experimental results in machine learning and NLP research.

Run lora in your codebase to provide a more convincing results.

Thank you for the advice. We ran LoRA in our training loop and observed that the reproduced results are very close to those reported in the paper. Therefore, the comparison is fair.

In Table 3, the adaptation either starts at layer 4 with a single hidden dimension or is applied to all layers with 8 hidden dimensions. Is there any reason why these specific options were chosen? (answer : to match the parameter count.)

Thank you for pointing this out! The former setup (NEAT-S) ensures a similar parameter count to FourierFT, which has the fewest parameters among the compared PEFT methods, allowing for a fair comparison. We began at layer 4 based on observations from AdaLoRA [2], which indicate that deeper layers are generally more important and allocating the parameter budget to these layers yields better results. Meanwhile, the latter setup (NEAT-L) ensures a similar parameter count to LoRA, which also uses a rank (hidden dimension) of 8.

the employed multi-layer network in NEAT, which is σ(σ(W0θ1)⋯θl−1)θl, also leads to low-rank output.

Thank you for your valuable question. Our contribution lies in proposing a nonlinear parameter-efficient fine-tuning framework. The benefit comes from the nonlinear approximation of model updates rather than high-rank approximation. We find that the model update is a complex function of $W^{0}$. Therefore, we introduce a nonlinear network that takes $W^{0}$ as input to approximate the update, providing an explicit method to achieve this. This enables NEAT to capture complex, nonlinear patterns in the weight space, thereby improving adaptation performance without increasing the number of parameters. Importantly, this architecture facilitates more efficient exploration of the optimization landscape, leading to better task adaptation, particularly in cases where linear methods like LoRA would require much larger ranks to achieve competitive results. We theoretically demonstrate that NEAT can achieve equal or greater expressivity compared to LoRA with fewer parameters. Furthermore, under our general framework, it also can be straightforward to achieve a high-rank update: $\sigma(\sigma(W^{0}\Theta_{1})\Theta_{2})$. However, we find that without the first activation function, NEAT has outperformed all baseline methods.

the authors did not provide sufficient effectiveness analysis

Thanks for your feedback. Here are some responses to your concerns:

1. Why does the nonlinear approximation of LoRA perform better than the original LoRA? This is analyzed from both theoretical and empirical perspectives in the paper. NEAT can model more complex weight updates, leading to a more expressive adaptation process.

2. The authors should perform an ablation study on LoRA and NEAT given the same value of r with different numbers of layers. It is important to note that in our experiments, we use the same value of $r$ for both NEAT and LoRA, resulting in the same parameter count (using vanilla NEAT). Therefore, our comparison is fair.

Table 1: parameter efficiency in MRPC

What is the computational cost NEAT should take to outperform LoRA and the other PEFT methods?

Thanks for pointing out the lack of computational cost analysis. First thing we want to note is that NEAT does not need the multiple intermediate layers to outperform LoRA. In fact, we use the vanilla NEAT in most parts of our experiments (besides some analysis).

Here we analyze the computational cost of vanilla NEAT (actual implementation we use for experiments!) and LoRA, running MRPC, STS-B tasks from GLUE benchmark and the fine-tuning stage of commonsense reasoning tasks. Kindly note that due to the different hyperparameter configuration across various tasks, the memory consumption varies across tasks.

Table2: hardware efficiency in MRPC:

Table 3: hardware efficiency in STS-B:

Table 4: hardware efficiency in Commonsense Reasoning:

It can be observed that the implementation we use for the experiments of NEAT matches the hardware efficiency of LoRA, with similar runtime and memory consumption (almost identical). For your convenience, we also include the data from multi-layer with 2 intermediate layers of NEAT in MRPC task below.

Table 5: hardware efficiency of multi-layer NEAT

It can further be noticed that even when introducing intermediate layers to form a small-scale neural network, the computational cost of NEAT is still comparable to LoRA, while achieving superior performance for the task!

The proof of proposition 5.1 is not closed. It only covers the equality case; however, the strict inequality is not yet established.

Thank you for your feedback. We have added additional details to complete the proof of Proposition 5.1 in the appendix. The key message of Proposition 5.1 lies in demonstrating the equivalence of expressivity between LoRA and NEAT. Specifically, our goal is to establish that both methods can achieve comparable levels of representation power under the assumptions outlined in the proposition, while NEAT may require fewer parameters. Thus, the focus is on equivalence rather than demonstrating a strict inequality between the two approaches.

Provide the parameter volumes (count) for the methods in Tab 1-2 for a fair comparison.

Thanks for pointing this out! We provide the parameter count in the LLM experiments below.

For commonsense reasoning tasks, we provide trainable parameters, typical runtime and computational overhead of the fine-tuning stage below.

Table 6: parameter efficiency of NEAT in Commonsense Reasoning:

It is evident that the parameter volumes for both methods are identical. Our proposed method, NEAT, does not introduce any additional trainable parameters. This ensures that NEAT maintains the same efficiency in terms of parameter count as LoRA, while achieving superior performance in the experiments.

One dataset is not enough for the study of depth. (Figure 2)

Thanks for your advice, we run this analysis on FGVC and DTD tasks as well to further illustrate our point! We will add the result in the revised version of the paper.

Table 7: multi-layer NEAT in FGVC

Table 8: multi-layer NEAT in Oxford-Pets

These results essentially demonstrate that the same phenomenon can generalize well to other tasks and is not a result of randomness. Note that increasing the depth will not always bring performance gain and this is also discussed in our paper.

Thanks for the response. Based on the author responses and the reviews from the other reviewers, my main concerns now lie in the comparison between LoRA and NEAT.

Given the formulation of LoRA as $W=\Delta W + AB$, the key point of NEAT is to approximate the low-rank matrix $A$ with the non-linear transformation of $\Delta W$. However, the authors have not provided sufficient supporting arguments on why NEAT can outperform LoRA. More actual justifications should be provided instead of 'efficient exploration of the optimization landscape' and 'capture complex, nonlinear patterns in the weight space'.

• Theoretically in proposition 5.1, under the same value of $r$, NEAT is not guaranteed to outperform LoRA. The proof of proposition 5.1 is still not closed in the revision. The authors only demonstrate the conditional equivalence between LoRA and NEAT, which can not prove the superiority of NEAT. The proved conditional equivalence of NEAT and LoRA is the minimum expectation. They share the same adaptation structure $AB$, which ensures the same computational requirements during inference. However, NEAT introduces additional computational overhead by approximating $A$ during fine-tuning. Therefore, the authors must provide stronger evidence to demonstrate that NEAT is indeed superior to LoRA. Without this justification, it is unclear why the additional complexity introduced by NEAT is warranted.
• Empirically, careful tuning for all baselines should be conducted. Also, the resource requirement during fine-tuning is important. I noticed that the authors sometimes compare LoRA and NEAT in the inference stage (same FLOPs, same number of parameters, etc.). Moreover, the ablation study on depth varies across different datasets, so the authors should repeat their experiments to improve the statistical reliability of the results.

I appreciate the authors' efforts during the rebuttal to address my concerns. However, given the current lack of proof (which would require further peer review), the need for tuning the baselines, and other issues, the manuscript still requires substantial revisions and does not yet meet the standards for acceptance.

Dear Reviewer vgTy,

Thank you for reviewing our paper and sharing your feedback. As the discussion phase is ending in three days, we want to check if you have any further concerns that we have not yet addressed.

We have made substantial revisions based on your comments and provided detailed responses to clarify and improve the work. If you feel our updates have resolved the issues you raised, we would appreciate it if you could consider adjusting your score accordingly.

If you have any specific points that still need clarification, please let us know, and we will address them promptly.

Best regards,

Authors

Thank you for your efforts. However, my concerns remain unresolved. The equations on L844 and L855 do not necessarily hold. The author should not rely on feedback to revise their theoretical proof during the rebuttal stage, especially when repeated modifications still fail to close it (currently, the appendix does not indicate changes made compared to the original manuscript). The manuscript still requires substantial revisions, further peer review and does not yet meet the standards for acceptance.

Could the authors provide an analysis of the typical runtime and computational overhead associated with NEAT?

Absolutely! Below, we provide the runtime and memory consumption for vanilla NEAT and LoRA on the MRPC, STS-B, and SST-2 datasets from the GLUE benchmark. Additionally, we compare NEAT and LoRA in terms of runtime and memory efficiency during the fine-tuning stage of commonsense reasoning tasks. We will include this analysis in the revised version of the paper. It can be observed that vanilla NEAT generally performs comparably to LoRA in terms of hardware efficiency, exhibiting similar runtime and nearly identical GPU memory consumption. Please note that due to different hyperparameter configurations across tasks, memory consumption may vary.

Table 1: parameter efficiency in MRPC

Table 2: parameter efficiency in STS-B:

Table 3: parameter efficiency in SST-2:

Table 4: parameter efficiency in Commonsense Reasoning:

For theoretical analysis, it may be worth considering that in many models, particularly transformers, Σ(d1l+d2l) can often be approximated as 2Σ(d2l), as sequential layers often satisfy d1l−1=d2l. From this perspective, NEAT might not offer significant theoretical advantages in expressivity as initially suggested.

Thank you for your feedback. While NEAT may not provide theoretical advantages when fine-tuning a nearly square matrix ($d_1 \approx d_2$), there are cases where NEAT offers improved parameter efficiency. For example, the first weight matrices of feed-forward layers often satisfy $d_2 \ll d_1$ [4,5]. In such scenarios, our theory suggests the improvement of NEAT over LoRA in parameter efficiency.

Lack of an ablation study isolating these changes (the introduction of non-linearity, the addition of a multiplicative W in the tunable component.)

Thanks for the insightful feedback! We conducted an ablation study using the vision tasks datasets below. In Section 6.2 of our paper, we show the proposed NEAT outperforms LoRA. From Table 6, we observe that both Nonlinear LoRA and Multiplicative LoRA underperform compared to NEAT. This demonstrates the effectiveness of introducing nonlinear approximation and explicitly using model weights as input to the nonlinear function.

Table 6: ablation on the changes NEAT introduces

[4] Vaswani, A. (2017). Attention is all you need. Advances in Neural Information Processing Systems.

[5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.

Thank you to the authors for their prompt response.

Regarding the analysis of typical runtime and computational overhead associated with NEAT: It would be helpful if the authors provided the total floating-point operations (FLOPs) instead of runtime. This is because the type and utilization rate of the GPU (which were not specified) can significantly influence the reported time. If the GPU is not under full load, methods with different computational overheads could exhibit similar runtimes. Including FLOPs would offer a more standardized and comparable metric.

Regarding the ablation study: Based on the authors' response to reviewer ZVEJ, it seems that these methods are highly sensitive to hyperparameter tuning. To demonstrate the effectiveness of the proposed NEAT method, I suggest that the authors carefully tune the hyperparameters for each method (at the very least, the learning rate, as LoRA is known to be particularly sensitive to it [1][2]). Additionally, the authors should provide detailed results under various configurations to ensure a fair and comprehensive comparison.

[1] Hayou, Soufiane, Nikhil Ghosh, and Bin Yu. "LoRA+: Efficient low-rank adaptation of large models." arXiv preprint arXiv:2402.12354 (2024). [2] Biderman, Dan, et al. "LoRA learns less and forgets less." arXiv preprint arXiv:2405.09673 (2024).

Non-linearity is inherently captured within the layers of a neural network. The need for additional non-linearity at this stage (PEFT adaptation) is not well justified. lack of motivation for introducing nonlinearity into lora.

Thank you for your feedback. We would like to clarify that the non-linearity is present in the model update. Equations (3) and (4) represent iterative model updates. The gradient of $L$ with respect to the model weight is a complex function of $W_0^{0}$, and the goal of PEFT is to approximate the weight update $\Delta W$. Although the transformer layers are nonlinear functions, they can not be used to capture the nonlinearity of the model update because they are frozen during fine-tuning. For the proposed NEAT, since $W_0^{0}$ is the input to this function, we aim to leverage a nonlinear network that takes $W_0^{0}$ as input to approximate the update directly. This approach allows us to efficiently capture more complex and richer transformations of the weights.

Theoretical analysis in Section 5 has limitations. The main issue is that it simply establishes an equivalence between a LoRA update and a single-layer factorization using a ReLU function. What does the proposition in Section 5 aim to establish?

Thank you for your feedback. In Proposition 5.1, our primary objective is to demonstrate the equivalence of expressivity between LoRA and NEAT. By establishing this equivalence in expressivity, we further claim that NEAT has the potential to improve parameter efficiency. This is because, in certain scenarios, the number of parameters required by NEAT to achieve the same level of expressivity could be fewer than those required by LoRA.

The method also appears sensitive to the choice of activation function and rank. More discussions about choosing an appropriate non-linearity function are needed.

Thanks for pointing this out! We conducted multiple analyses using different activation functions on the StanfordCars dataset and will include additional datasets in the revised version of the paper to provide more insights on this topic. For the simplicity of presentation, we only show the results where the learning rate for the classification head is fixed to 1e-2 and the learning rate for the backbone (b_lr in the table) (i.e. the adapters) is in a range of {8e-3, 5e-3, 3e-3}.

Table 1: results on StanfordCars using various activations

From these results, we conclude that the choice of activation function influences performance to some extent. For example, the sinusoidal activation function yields superior performance when using a small learning rate, whereas activation functions like GELU, Tanh, and ReLU show decreasing performance as b_lr decreases.

However, based on our experience with NEAT, a simple ReLU function, in most cases, performs well enough to match the performance of LoRA or even surpass it. Additionally, the choice of activation function can often be simplified by using ReLU.

Thank you for your response. I’d like to highlight a few key points for clarification and further discussion.

Firstly, your method lacks a framework for approximating the full chain of updates leading to $\Delta W^T$ (where $T$ is the final optimization step). Instead, it relies solely on a one-step approximation of the update in a low-rank setting. However, neither the paper nor this response provides any justification for why introducing a nonlinear function into LoRA leads to a meaningful approximation of $\Delta W^T$.

Regarding Proposition 5.1, you mention expressivity but do not elaborate on what you mean by this concept or how it is addressed in the proposition. Could you clarify what “expressivity” refers to in this context and point out which parts of the proposition substantiate this?

Finally, I think the main issue isn’t just demonstrating that your method performs comparably to vanilla LoRA. Rather, it’s important to provide a stronger justification for why and when the introduced nonlinearity is beneficial. For instance, under what circumstances do specific nonlinearities (e.g., ReLU, sine, tanh) result in improved performance? Are there task-specific properties or characteristics that determine which nonlinearity is more effective?

I appreciate the new results. Do you have the comparative LORAs in these cases? It seems it is very sensitive to hyperparameters. When you say b_lr, do you mean the learning rate for LORAs or the original weights of the model?

I'd also like to ask why ReLU is the one working best for this method since it basically means the negative gradients are not used for updating the model weights. Could you comment on that, please?

Firstly, your method lacks a framework for approximating the full chain of updates leading to $\Delta W$ (where $T$ is the final optimization step). Instead, it relies solely on a one-step approximation of the update in a low-rank setting. However, neither the paper nor this response provides any justification for why introducing a nonlinear function into LoRA leads to a meaningful approximation of $\Delta W$.

Thanks for your feedback. Our proposed method aims to approximate the full chain of updates. From Eqn. 3, we observe that $W_1^{0}$ is a function of the model weight $W^{0}$ and $W_2^{0}$ depends on $W_1^{0}$. Therefore, $W_2^{0}$ forms a composite function of $W^{0}$. Extending this pattern, $W_t^{0}$ becomes a composite function of $W^{0}$. This implies that $\Delta W$ is also a composite function of $W^{0}$. Given the complexity and nonlinearity of this composite function, we propose introducing a nonlinear network with $W^{0}$ as the input to approximate it. Through iterative updates, the nonlinear network progressively approximates the full chain of updates.

Regarding Proposition 5.1, you mention expressivity but do not elaborate on what you mean by this concept or how it is addressed in the proposition. Could you clarify what “expressivity” refers to in this context and point out which parts of the proposition substantiate this?

Thank you for your valuable feedback. The term `expressivity’ in Proposition 5.1 refers to the capacity of a model to minimize the loss function, which is commonly used in the literature of machine learning theory [1,2,3]. Specifically, the proposition compares the achievable loss for different configurations of NEAT and LoRA, illustrating their relative expressive power. Proposition 5.1 makes the following claims, which involve three terms representing the minimum attainable loss:

1. NEAT with $2r$ hidden dimensions: $\min_{\substack{\Theta_1 \in \mathbb{R}^{d_2 \times 2r},\Theta_2 \in \mathbb{R}^{2r \times d_2}}} \mathcal{L}(\mathcal{D}_{\text{train}}; W^0 + f(W^0; (\Theta_1, \Theta_2)))$
2. rank-$r$ LoRA: $\min_{\substack{A \in \mathbb{R}^{d_1 \times r},B \in \mathbb{R}^{r \times d_2}}} \mathcal{L}(\mathcal{D}_{\text{train}}; W^0 + A B)$
3. NEAT with $r$ hidden dimensions: $\min_{\substack{\Theta_1 \in \mathbb{R}^{d_2 \times r},\Theta_2 \in \mathbb{R}^{r \times d_2}}} \mathcal{L}(\mathcal{D}_{\text{train}}; W^0 + f(W^0; (\Theta_1, \Theta_2)))$.

The proposition asserts that: NEAT achieves the same expressivity as LoRA, while NEAT requires much less parameters, and therefore NEAT is more parameter efficient. Specifically, our results show that the minimum attainable loss using rank-$r$ LoRA can also be achieved by NEAT with $2r$ hidden dimensions Conversely, the minimum attainable loss by NEAT with $r$ hidden dimensions can also be achieved by rank-$r$ LoRA. This also implies that the function classes realized by NEAT with $O(r)$ hidden dimensions and rank-$r$ LoRA are equivalent in terms of their expressivity, since the result holds for any loss functions. We will incorporate this clarification in our revised manuscript to ensure the concept of `expressivity’ and its substantiation in Proposition 5.1 are clear. In conclusion, the above argument implies an improved parameter efficiency achieved by NEAT (also pointed out at the beginning of page 5 in our paper). Since (a) NEAT with $2r$ dimensions has $2rd_2$ parameters whereas rank-$r$ LoRA has $r(d_1+d_2)$ parameters, and (b) $d_2$ can be much smaller than $d_1$, our theoretical result demonstrates a significant improvement in parameter efficiency in this setting.

[1] Raghu, M., Poole, B., Kleinberg, J., Ganguli, S., & Sohl-Dickstein, J. (2017, July). On the expressive power of deep neural networks. In international conference on machine learning (pp. 2847-2854). PMLR. [2] Gühring, I., Raslan, M., & Kutyniok, G. (2020). Expressivity of deep neural networks. arXiv preprint arXiv:2007.04759, 34. [3] Zeng, Y., & Lee, K. (2023). The expressive power of low-rank adaptation. arXiv preprint arXiv:2310.17513.

Dear Reviewer ZVEJ,

Thank you for your thoughtful review of our paper and your insightful feedback. We hope this message finds you well.

As the discussion phase will end in three days, we wanted to check if there are any remaining concerns or questions we can address. We have carefully revised our paper based on your comments and provided detailed responses to your points. If the revisions and responses have resolved your concerns, we kindly hope you might consider re-evaluating your score.

If you have any additional feedback or suggestions, we would be happy to address them promptly. Your input is greatly valued, and we are committed to improving our work based on your recommendations.

Thank you once again for your time and support.

Best regards,

Authors

Dear Reviewer ZVEJ,

Thank you for your thoughtful feedback and for engaging with our work. With just one day remaining in the author-reviewer discussion period, we would like to kindly request your review of our responses to your comments.

We hope our clarifications and revisions address your concerns effectively. If you find them satisfactory, we would greatly appreciate it if you could consider updating your rating. Please do not hesitate to reach out if you have any further questions or require additional clarification before the discussion period ends.

Best regards,

Authors