Curvature Tuning: Provable Training-free Model Steering From a Single Parameter

We thank the reviewer for the constructive feedback! Before addressing specific concerns, we briefly restate the central aim of our paper: to establish CT as a highly efficient approach to parameter tuning, which is orthogonal and complementary to established parameter-centric PEFT methods. In our experimental evaluation, we achieve results that are comparable to other PEFT methods, but with considerably less parameters.

Local Curvature

We thank the reviewer for suggesting an interesting theoretical comparison: understanding the impact on the approximation error of increasing model size vs tuning curvature of a fixed-size model. In the following, we discuss the implications of Theorem 3.1 for fixed-size models, and leave further theoretical analysis to future work. The proposed CT method is grounded in the theory of Affine Spline Operators (ASOs). By tuning the curvature of each activation function, CT allows to increase local expressivity of a model (i.e. local curvature). Importantly, while the hypothesis space of ASOs parameterized by sufficiently deep ReLU networks enjoys universal approximation ability [1], Th 3.1 shows that once a model size is fixed, affine spline operators are characterized by zero (i.e. vanishing) local curvature a.e. in their domain. Importantly, to approximate functions of non-zero local curvature, increasing model size of ASOs provably yields non-zero error in the non-asymptotic limit, due to their local affine nature. This is further discussed in the motivating example in Section 3, as well as Appendix C.2.

Interpretability of CT

As acknowledged by reviewers cxvD and W2ME, CT is interpretable in that it exposes two learnable parameters per neuron, which provably modulate the activation function’s curvature. In turn, this controls the curvature of decision boundaries, which is intimately tied to both generalization and robustness [2,3]. To see this, we note that for a deep network $f$ , the decision boundary of any class $i$ and $j$ is given by the set of $\{ x \in R^d : g(x) := f_i(x) − f_j (x) = 0 \}$, for $i \ne j$. Particularly, $g$ is itself a deep network, sharing the same parameters as $f$ up until the penultimate layer. Hence, modulating non-linearity of activation functions via $\beta$ directly controls the curvature of both model function and its decision boundaries, which is provably connected to generalization and robustness [2,3]. We are happy to add a paragraph to the main paper clarifying this relationship and its connection to interpretable curvature tuning.

Performance on Transformers

The paper posits two potential reasons for the underperformance of Trainable CT vs LoRA on ReLU Swin-T/S models pretrained on Imagenette:

1. Trainable CT uses significantly fewer parameters than LoRA (0.71% and 0.58% on Swin-T/S).
2. The small size of the Imagenette pretraining dataset results in weaker representations.

To investigate the second factor, we conducted an additional experiment using ImageNet-pretrained Swin-T/S models, with GELU activations (equivalent to CTU with $\beta=0.6403$ and $c=1$).

Results: Trainable CT achieves an average relative improvement over the frozen baseline of 2.92% (Swin-T) and 3.76% (Swin-S). The gap with LoRA is considerably narrower (3.19% and 5.42%) than that observed in the ReLU experiment on Imagenette. This supports our second hypothesis: the earlier performance gap was partly due to weaker representations from small-scale pretraining, which amplified the advantage of LoRA’s higher parameter count. These new results suggest that Trainable CT can achieve competitive performance while using only 0.71% and 0.58% of LoRA’s trainable parameters, underscoring its strong parameter efficiency. Importantly, CT and LoRA operate from orthogonal perspectives and can be combined for improved performance—our goal is not to outperform LoRA directly, but to offer a complementary PEFT method. Additionally, this experiment showcases the generality of Train CT: which covers GELU-based models.

CT vs. Higher-rank and Tuned-$\alpha$ LoRA

We extend our LoRA comparison to higher ranks and tuned $\alpha$ values on ImageNet-pretrained ResNet-18 and Imagenette-pretrained Swin-T (ReLU-based). More specifically, we test LoRA with $r \in {1, 2, 4}$ and $\alpha \in {r, 2r, 4r}$ and report the best performance.

Results: On ResNet-18, Train CT outperforms LoRA with rank 1/2/4 by 10.20%, 6.63%, and 3.22%, respectively, indicating that CT remains highly effective even when compared to stronger LoRA baselines. While for Swin-T, Trainable CT trails LoRA (rank 1/2/4) by 8.29%, 16.30% and 21.17%, respectively.

Setting $c=0.5$ for CTUs

The rationale for choosing $c = 0.5$ is explained in the last paragraph of Sec 3.1. Since the SiLU and SoftPlus tend to shift the mean of the activations in opposite directions (negative for SiLU and positive for SoftPlus), setting $c = 0.5$ balances these effects and helps mitigate the shift, making the mean output closer to ReLU.

To demonstrate why this is beneficial, we conducted an additional ablation comparing Steering CT vs SiLU- and SoftPlus-only variants (both reparameterized by CTU) on ResNet-18 and ResNet-50:

Results: The SiLU version trails steering CT by 1.98% and 0.94% on ResNet-18 and ResNet-50, respectively. And the SoftPlus version trails steering CT by 0.53% and 0.54%, respectively. These results highlight the performance degradation caused by the output shift in simpler formulations, and the effectiveness of balancing the two. Additionally, we observed that the average optimal $\beta$ for CT is 0.84 (ResNet-18) and 0.94 (ResNet-50), while for SiLU and SoftPlus, the values are closer to 1—0.99/0.95 (SiLU) and 0.94/0.98 (SoftPlus)—suggesting more limited flexibility for steering. This provides further support for the necessity of setting $c = 0.5$, enabling broader adaptability.

Applicability of CT in More Contexts

At its core, CT provides a mechanism for steering model behavior and is not tied to any specific modality. The steering CT can be applied directly in any domain via a lightweight grid search procedure, while Trainable CT only requires a task-specific objective, similar to other PEFT methods such as LoRA. Given this flexibility, we believe CT can be readily extended to domains beyond vision. While we focused on image classification in this paper, we agree that applying CT to language model finetuning (e.g., for GRPO) is an exciting direction for future work.

References

[1] Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activations. Hanin. Mathematics. 2019.

[2] Spectrally-normalized margin bounds for neural networks. Bartlett, et al. NeurIPS 2017.

[3] Robustness via Curvature Regularization, and Vice Versa. Moosavi-Dezfooli, et al; CVPR, 2019

Thank you for the detailed response to my review.

We are happy to add a paragraph to the main paper clarifying this relationship and its connection to interpretable curvature tuning.

Please, I think this would aid the manuscript.

I am not convinced by the results under the heading 'Performance on Transformers'. Quite often the performance seems to degrade versus the frozen model.

In my original review, I said:

Invalid baselines: the benefit of Train CT over LoRA is deceptive, since LoRA is only rank 1, and has not been tuned.

While it is good to see more thorough LoRA results, I'm still not convinced by the results. It would be simpler to take some existing results, or cite a thorough methodology, e.g. [1]

[1]: LoRA+: Efficient Low Rank Adaptation of Large Models

CT and LoRA operate from orthogonal perspectives and can be combined for improved performance

Are you able to show that the methods can be combined, and that an improvement is obtained?

Overall, I appreciate the clarifications given by the authors regarding my questions. Unfortunately I am still unconvinced by the results: (a) the results are only on image problems, (b) the method isn't applicable with general activation function, (c) there is no comparison/ablation with a 'dumb' baseline (e.g. we add the most obvious learned parameter that we can, e.g. relu(\beta x)), (d) Trainable CT reasonably often degrades performance.

We thank the reviewer for engaging in the discussion, and for providing detailed feedback. Before replying to individual points, we would like to reiterate the core design philosophy and advantages of Curvature Tuning.

1. The proposed method strives for extreme parameter efficiency, whereby up to two parameters are injected per activation neuron. As such, rather than focusing on beating SOTA on finetuning, our experiments center around evaluating downstream generalization and robustness in settings with constrained parameter counts, whereby we strive to provide the most parameter efficient method.
2. Curvature Tuning operates orthogonally to parameter-centric adaptation methods, in that it modulates the curvature of activation functions by changing the functional form of their nonlinearity. In line with our parameter efficiency goal, CT takes advantage of the functional form of ReLU, and proposes a flexible reparameterization whose hypothesis space encompasses several popular activations, including GeLU, SiLU, and Softplus, making it broadly applicable to modern finetuning settings.
3. Thanks to the theoretical connection with Affine Spline Operators (Sec 3), the method provably modulates curvature, providing considerable benefits on downstream robustness benchmarks. By modulating activation functions, the paper opens a novel avenue for parameter-efficient finetuning.

In the following, we reply to each raised point.

Target Activation Functions

While it is in principle possible to design activation-agnostic methods, e.g. via Taylor expansions, this approach would violate one of the core design principles of Curvature Tuning, namely to inject as few parameters as possible into the model.

By taking advantage of the functional form of ReLU and its smoothed versions, we propose a novel reparameterization that encompasses several popular activation functions, including ReLU, SiLU/Swish, GeLU and Softplus.

In fact, what on the one hand the method may lose in terms of generality, is on the other hand gained in terms (1) explicit curvature modulation and (2) parameter efficiency.

In stark contrast, parameter-centric adaptation methods cannot operate on activation functions and the associated non-linearity, and entail non-interpretable design choices, such as adapter placement and rank.

Choice of Baselines

We note that the proposed naive baseline, ReLU($\beta x$), falls in the realm of parameter-centric adapters, which as discussed above are unable to change the functional form of the non-linearity. An alternative, simple, naive baseline is expressible by CTUs. As discussed in Sec 3, for $c=1$ and $\beta = 0$, one obtains $x \mapsto \frac{1}{2} x$ which is a linear function with zero curvature globally. Such a baseline is naive in that it offers maximum curvature regularization but limited expressivity, and is orthogonal to parameter-centric methods.

Importance of $r=1$ LoRA Baseline

We would like to briefly justify and clarify the choice of low rank for the LoRA adapters. Since Curvature Tuning operates orthogonally (and thus complementarily) to parameter-centric finetuning methods, our experimental evaluation was designed to showcase the method’s efficiency in budget constrained settings. Where CT injects up to two parameters per activation neuron, LoRA with $r=1$ and no biases uses at least twice the parameter count, making it a fair baseline w.r.t. efficiency.

Performance on GeLU Transformers

We thank the reviewer for pointing out that Trainable CT sometimes underperforms the frozen model. After careful inspection, we found that the learning rate inherited from the ReLU-based Swin-T/S setup in our original submission was too high for the GeLU configuration. We provide below an updated evaluation on GELU-based Swin-T/S experiments, whereby we perform cross-validation to select the optimal learning rate for each method (i.e., the baseline, Trainable CT, and LoRA). We report the best results for each method below:

Results: Trainable CT outperforms the baseline on 9 out of 12 datasets for Swin-T, and on 11 out of 12 datasets for Swin-S.

We thank the reviewer for their thorough and encouraging feedback! In response, we conducted an additional experiment and provided detailed clarifications addressing the reviewer’s concerns. Before delving into specific points, we would like to reiterate the central aim of our paper: to establish CT as a highly efficient approach to parameter tuning, which is orthogonal and complementary to established parameter-centric PEFT methods. In our experiments, CT achieves competitive performance with significantly greater parameter efficiency.

Weakness 1: Limited Architectural Diversity in Evaluation

To demonstrate CT's applicability to broader model families, we conducted an additional experiment applying Trainable CT to an ImageNet-pretrained VGG11 (with batch normalization). We compared it with linear probing and LoRA ($r=1$, $\alpha=1$):

Results: Trainable CT achieves an average relative improvement of 5.55% over linear probing and 3.44% over LoRA, demonstrating strong performance even on architectures not originally considered in the paper.

Weakness 2: Incomplete Robustness Evaluation

We omitted LoRA from the robustness experiments because applying it to improve robustness would introduce confounders compared to Steering CT. Indeed, methods such as LoRA require an explicit optimization target—such as minimizing cross-entropy loss for classification tasks. In contrast, Steering CT can improve robustness by simply adjusting the curvature of the model decision boundary, without relying on labeled data or explicit loss functions. Similarly, while Trainable CT directly modulates the curvature of decision boundaries (and thus bears a direct effect on adversarial robustness [1]), a direct and fair comparison with LoRA would require incorporating adversarial training techniques, such as generating adversarial examples using PGD, which would somewhat obfuscate the isolated effect of Trainable CT.

Beyond the above experiment, deriving methods specifically tailored around CT that enable more targeted defenses against distributional shifts or adversarial threats is an exciting direction for future work.

Question 1: Limited Architectural Diversity in Evaluation

See Weakness 1.

Question 2: Incomplete Robustness Evaluation

See Weakness 2.

Question 3: Limited Experimental Scope

CT is not limited to vision tasks. The steering CT can be applied directly in any domain via a lightweight grid search procedure, while Trainable CT only requires a task-specific objective, similar to other PEFT methods such as LoRA. Therefore, we anticipate that CT is applicable to other domains, including NLP, multimodal learning, and reinforcement learning. Exploring these directions would be an intriguing avenue for future research.

Question 4: Additional Clarification on Class Imbalance

We do include class-imbalanced datasets in our experiments—most notably DermaMNIST, where the most frequent class accounts for 67% and the least for only 1% of training samples. As shown in Table 1 and Appendix Table 4 of our paper, Trainable CT consistently outperforms all considered baselines on this dataset across ResNet-18/50/152. This suggests that CT maintains robustness under skewed label distributions.

We sincerely appreciate the reviewer’s thoughtful and constructive comments. We hope that our responses and the additional experiment have adequately addressed the raised concerns and further highlighted the strengths and generality of our method.

References

[1] Robustness via Curvature Regularization, and Vice Versa. Moosavi-Dezfooli, et al; CVPR 2019.

To further showcase the direct impact of curvature tuning on adversarial robustness, we extend the RobustBench evaluation to Trainable CT and LoRA, each finetuned in the vanilla setting, without introducing adversarial examples. This experiment better elucidates the direct impact of each method on robustness, without introducing confounders such as training with adversarial samples or external regularization.

Specifically, we transfer ImageNet-pretrained ResNet-18/50/152 models to CIFAR-10/100 using the same setup as in our main paper—linear probing (i.e., Frozen in the table), Trainable CT, and LoRA . We then evaluate the adversarial robustness of the resulting models under an $\ell_\infty$ attack using RobustBench. The results are as follows:

Results: On ResNet-18/50/152, Trainable CT achieves an average relative improvement of 420.00% / 1150.00% / 750.00% over linear probing. In stark contrast, LoRA provides a slight boost in robustness, while at times has detrimental effects with 130.00% / 83.33% / –66.67%, respectively. These results indicate that even without explicit adversarial training, Trainable CT can substantially enhance $\ell_\infty$ adversarial robustness, since it directly modulates decision boundary curvature, whereas LoRA offers limited or even negative improvements, since it leaves activation function nonlinearity unchanged. This empirical finding further supports the practical advantage of Trainable CT w.r.t. robustness: by directly modulating the curvature of decision boundaries, it provides direct improvements on adversarial robustness, even without explicit adversarial training. We are happy to include the above evaluation as two additional columns in Table 2 in the main paper.

We thank the reviewer for their thorough and encouraging feedback. In response, we conducted two additional experiments and provide below detailed clarifications to address the reviewer’s concerns.

Before addressing the raised concerns, we begin by reiterating the central aim of our paper: to establish Curvature Tuning (CT) as a highly efficient approach to parameter tuning, which is orthogonal and complementary to established parameter-centric Parameter-Efficient Fine-Tuning (PEFT) methods. In our experimental evaluation, we achieve results that are comparable to other PEFT methods, but with considerable gains in terms of parameter-efficiency and novel applications to downstream robustness.

Weakness 1: Limited Utility on Other Activations

We agree that our theoretical guarantees are currently tied to ReLU-based networks. However, this does not preclude the application of CT to other activations. In fact, SiLU and SoftPlus are special cases of our Curvature Tuning Unit (CTU), and Trainable CT can be directly applied to models using these activations. Moreover, for GELU—commonly used in transformer models—it can be approximated by SiLU with a CTU parameterization of $\beta=0.6403$ and $c=1$. This allows us to apply Trainable CT to GELU-based models by initializing with these values. To address this point concretely, we conducted new experiments applying Trainable CT to ImageNet-pretrained Swin-T and Swin-S models using their original GELU activations (unlike our earlier experiments, which replaced GELU with ReLU). We compare against linear probing and LoRA ($r=1$, $\alpha=1$):

Results: Trainable CT improves over the frozen baseline by 2.92% (Swin-T) and 3.76% (Swin-S) on average, while trailing LoRA by only 3.19% and 5.42%, respectively—despite using only 0.71% and 0.58% of LoRA's trainable parameters.

Weakness 2: Underperformance in Transformers

In the original submission, we hypothesized two potential reasons for the underperformance of Trainable CT compared to LoRA on ReLU-based Swin-T/S models pretrained on Imagenette:

1. Trainable CT uses significantly fewer parameters than LoRA (0.71% and 0.58% on Swin-T/S).
2. The small size of the Imagenette pretraining dataset results in weaker representations, amplifying the benefit of high-parameter methods like LoRA.

In our new experiment on ImageNet-pretrained, GELU-based Swin-T/S models as shown above, the number of trainable parameters for both Trainable CT and LoRA remains identical to the original experiment. Thus, the first factor still holds—CT continues to operate with a significantly smaller parameter budget. Importantly, the new results support the second hypothesis: when models are pretrained on a much larger dataset (ImageNet), the performance gap between Trainable CT and LoRA narrows. This suggests that the earlier underperformance was at least partly due to the limited representation capacity of models pretrained on the smaller Imagenette dataset. In summary, as demonstrated by the new experiment, while Trainable CT still trails LoRA by 3.19% (Swin-T) and 5.42% (Swin-S), it achieves this competitive performance with only 0.71% and 0.58% of LoRA's trainable parameters—highlighting its strong parameter efficiency and practical value.

Weakness 3: Tuning Overhead of Steering CT

We acknowledge that the non-trainable version of CT involves a grid search over $\beta$. However, we observe that the optimal $\beta$ values tend to be close to 1, suggesting that the search range can be substantially narrowed in practice (e.g., to [0.9, 1]) for efficiency. Moreover, popular PEFT methods such as LoRA also require hyperparameter tuning—for instance, selecting appropriate values for rank and $\alpha$—and typically involve grid searches as well. Thus, tuning overhead is not unique to our method.

Question 1: U-shaped Distributions of $\beta$ and $c$

As noted in Footnote 5 of the paper, the U-shaped distributions may be partially attributed to the sigmoid-based parameterization of $\beta$ and $c$ (which constrains them to [0,1]). Interestingly, the average values of these parameters are close to the optimal settings discovered in the steering version of CT, which we consider a desirable outcome. We have not explicitly regularized these distributions, as it's unclear whether such U-shapes are beneficial or detrimental. However, we agree this is an intriguing direction and plan to explore regularization effects in future work.

Question 2: Negative Impact of the Shift

To demonstrate the value of our current CTU formulation in mitigating the shift caused by simpler formulations, we compare the steering version of CT against the simpler formulation of just using SiLU or SoftPlus (both reparameterized as in CTU) on ResNet-18 and ResNet-50:

Results: The SiLU-only version trails steering CT by 1.98% and 0.94% on ResNet-18 and ResNet-50, respectively. And the SoftPlus-only version trails steering CT by 0.53% and 0.54%, respectively. This performance gap illustrates the negative impact caused by using simpler formulations. We also observed that the average optimal $\beta$ for CT on ResNet-18/50 is 0.84/0.94. In contrast, for the other two formulations, the values are closer to 1—0.99/0.95 for SiLU and 0.94/0.98 for SoftPlus—indicating a smaller room for model steering. This reflects the negative impact of the shift from another perspective.

Question 3: CT Performance with Higher Parameter Budget

In our current setup, Trainable CT already uses the maximum number of parameters allowed under its formulation—specifically, by assigning one pair of parameters ($\beta$, $c$) to each output neuron of the activation functions throughout the network. Therefore, it is not possible to increase the parameter count within the current formulation. However, since Trainable CT is orthogonal to other PEFT methods, any additional parameter budget can be leveraged by combining it with methods like LoRA. This opens up a promising direction for achieving even stronger performance while maintaining modularity and efficiency.

We sincerely appreciate the reviewer’s thoughtful and constructive comments. We hope that our responses and additional experiments have sufficiently addressed the concerns raised and clarified the strengths and contributions of our work.

We thank the reviewer for their thorough and encouraging feedback! As suggested, we have conducted three additional experiments to further demonstrate the effectiveness and generality of Curvature Tuning (CT). These experiments address broader model families, activation functions, and comparisons with additional LoRA configurations.

We also take the opportunity to emphasize that the aim of the paper is to establish CT as a highly efficient approach to parameter tuning, which is orthogonal and complementary to established parameter-centric PEFT methods. In our experimental evaluation, we achieve results that are comparable to other PEFT methods, but with considerable gains in terms of parameter-efficiency.

Experiment 1: CT on VGG11

To evaluate the effectiveness of CT on a different convolutional architecture, we apply Trainable CT to an ImageNet-pretrained VGG11 model (with batch normalization). We compare it against linear probing and LoRA with $r=1$ and $\alpha=1$.

Results: Trainable CT achieves an average relative improvement of 5.55% over linear probing and 3.44% over LoRA, showing its strong performance even on models not originally considered in the paper.

Experiment 2: CT on GELU-based Swin-T/S

To test CT’s compatibility with other activation functions, we begin with GELU as a representative case. Since GELU can be approximated by SiLU, which is parameterized by CTU with $\beta=0.6403$ and $c=1$, we can apply Trainable CT to GELU-based models by starting with these values. More specifically, we apply Trainable CT to ImageNet-pretrained Swin-T and Swin-S, using the original GELU activations (unlike our earlier experiment in the paper, which replaced GELU with ReLU). And we compare it with linear probing and LoRA ($r=1$, $\alpha=1$).

Results: Trainable CT achieves an average relative improvement over the baseline of 2.92% (Swin-T) and 3.76% (Swin-S). While Trainable CT still trails LoRA by 3.19% and 5.42% respectively, the performance gap is notably smaller than in the original paper. This supports our hypothesis in the original paper that the larger gap in the earlier ReLU-based Swin experiments (pretrained on the smaller Imagenette) stemmed from underpowered representations, which gave LoRA’s greater parameter count a stronger advantage. On ImageNet-pretrained models now, Trainable CT achieves more comparable results despite using just 0.71% and 0.58% of LoRA’s trainable parameters.

We also emphasize: CT is complementary to LoRA, not a replacement—beating LoRA is not our goal, but rather demonstrating CT's efficiency and compatibility.

Experiment 3: CT vs. Higher-rank and Tuned-$\alpha$ LoRA

We extend our LoRA comparison by evaluating higher ranks and tuned $\alpha$ values on ImageNet-pretrained ResNet-18 and Imagenette-pretrained Swin-T (ReLU-based). More specifically, we test LoRA with $r \in {1, 2, 4}$. For each rank, we use $\alpha \in \{r, 2r, 4r\}$ and report the best performance.

Results: On ResNet-18, Trainable CT outperforms LoRA with rank 1/2/4 by 10.20%, 6.63%, and 3.22%, respectively, indicating that CT remains highly effective even when compared to stronger LoRA baselines. While for Swin-T, Trainable CT trails LoRA (rank 1/2/4) by 8.29%, 16.30% and 21.17%, respectively.

Future Work: CT on Language Model

As a promising direction for future work, we plan to evaluate Trainable CT on language models and benchmarks. This would allow us to examine CT's effectiveness in non-vision domains.

Conclusion

These new experiments collectively reinforce the generality and practicality of Curvature Tuning across several key aspects suggested by the reviewer:

• More Models: Demonstrated in Experiment 1 (VGG11) and Experiment 2 (GELU-based Swin-T/S), showing that CT applies to broader model families.
• More Activation Functions: Experiment 2 illustrates CT's applicability to GELU-based networks, beyond the original ReLU-based settings.
• More Finetuning Techniques: Experiment 3 highlights CT’s competitiveness even against stronger LoRA configurations with higher rank and tuned alpha.

We appreciate the reviewer’s insightful suggestion and hope these additions further validate CT as a compelling and complementary approach to parameter-efficient finetuning.

We thank the reviewer for supporting our submission!