Generating Informative Samples for Risk-Averse Fine-Tuning of Downstream Tasks

We sincerely thank the reviewer for the thoughtful and constructive feedback. For clarity, we refer to individual points using Q (Questions) and W (Weaknesses), e.g., Q1 refers to the first bullet under Questions, W1 to the first under Weaknesses.

---

[R3Q1] (Q1, W1) . Consider taking $\theta_0 = 0$... Can the authors explain why the samples are guided using only the initial point? Why aren't samples refined with new predictors? /... Maybe what the authors missed to highlight is that their procedure should only be used after a first round of empirical risk minimization?

[R3A1] Thank you for raising this point. We have addressed this question in conjunction with a related question from Reviewer dRNj in [R2Q2] and its corresponding response [R2A2]. Kindly refer to [R2A2] for our detailed explanation.

---

[R3Q2] (Q2, W6, W2, W5) Eq; 8: the bound on square gradient norms does not give a bound on the variance... / Can the authors prove the reduced variance they hope their method can bring (on both losses and subgradients)? / The key part of the algorithm is in having access to low variance unbiased estimates of the loss and the subgradients of the CVAR objective ... Their argument goes through proving the convergence of the algorithm by some means and with some bounds from which they argue what should be the choice of initial distribution. This circumvolved path hides whether the claimed objective of the authors "getting low variance oracles for optimization" is reached or not.... / ... it would be best for the results to show e.g. lower variance of the stochastic estimates.

[R3A2] We appreciate the reviewer’s insightful comments.

First, we clarify that our method targets reduction of the overall stochastic noise upper bound $\hat{v}(q)$, rather than explicitly minimizing the variance of per-iteration subgradients or per-iteration loss estimates. Our primary goal of designing importance sampling distribution is to minimize the distance between the final trained parameter and the true optimum, as stated in Theorem 1.

Regarding (8), it is introduced to justify our use of an importance sampling distribution proportional to the pretrained model's loss. While classical variance reduction analyses for SGD often optimize sampling proportional to the gradient norm squared (Zhao & Zhang, 2015), Eq. (8) shows that the gradient norm is bounded both above and below by a linear function of the loss under the PL condition. This supports our surrogate choice of the pretrained model loss as a practical proxy for gradient magnitude. In other words, we do not claim that we minimize the gradient squared norm's upper bound, instead, Eq (8) shows the proportional relationship between the gradient norm and the loss value.

The reason we use the term “variance” following Eq. (8) is that, by taking expectation over the inequality, we obtain a second-order moment of the gradient norm. And the expectation of the pretrained model’s loss appears in $\hat{v}(q)$ with the optimal importance weight function $w*$ (substitution of $w^*$ into $\hat{v}(q)$). We agree that referring to this as “variance” is misleading, and we will revise the statement at line 230 to: “high-loss samples contribute proportionately to the squared norm of the gradient.”

That said, we fully agree that demonstrating subgradient variance reduction is important. To fully address the reviewer's concern, we provide additional analysis regarding the subgradient and loss. Theorem 1 and Appendix Eq. (20), inherently address this. The subgradient of the $k$-th iteration objective is denoted $g_k$, and its squared norm is bounded as: $\Vert g_k \Vert^2 \le v(x)^2$ where $v(x) = \sqrt{ \frac{1}{(1-\beta)^2}\Vert \frac{p(x_k)\partial \ell (\theta_k;x_k)}{q(x_k)} \Vert^2 + 1 +\frac{p(x_k)^2}{q(x_k)^2 (1-\beta)^2} }$. Taking expectation over $q$, on the left hand side, we have the second moment of the subgradient, $\mathbb{E}[\Vert g_k \Vert^2]$ . In (23)-(27), (line 903-906, in Appendix), it directly shows that the stochastic noise $\hat{v}(q)$ is an upper bound of the $\mathbb{E}[\Vert g_k \Vert^2]$ and $\hat{v}(q)$ is minimized by the importance sampling distribution with weight $w^*$, as stated in Theorem 1.

For per-iteration loss variance reduction, we consider a $\theta_k$, the model parameter at the $k$-th iteration and corresponding dual form objective is given as $F_{\beta}(\theta_k, \alpha_k) = \alpha_k + \frac{1}{1-\beta}\mathbb{E}[(\frac{p(x)}{q(x)}(\ell(\theta_k;x)-\alpha_k)^{+})]$. Then it is straightforward to see that the variance of this estimate is reduced if: $\mathbb{E}[(\frac{p(x)}{q(x)})((\ell(\theta_k;x)-\alpha_k)^{+})^2] < \mathbb{E}[((\ell(\theta_k;x)-\alpha_k)^{+})^2]$. Considering the pretrained model loss-based importance sampling distribution $q(x) \propto \varphi(\ell(\theta_0;x))p(x)$, we have $\mathbb{E}[\frac{Z((\ell(\theta_k;x)-\alpha_k)^{+})^2}{\varphi(\ell(\theta_0;x))}] < \mathbb{E}[((\ell(\theta_k;x)-\alpha_k)^{+})^2]$ where $Z$ is the normalizing constant. (Informal statement: Intuitively, the result suggests that if instances with high loss under $\theta_0$ continue to have high loss during fine-tuning (i.e., $\theta_k$), then the importance sampling concentrates on high-loss regions, thereby reducing variance. As discussed in [R2A3], this assumption can be reasonable in the context of fine-tuning from a well-initialized model, and high-loss-inducing samples often require multiple optimization steps to be effectively addressed in risk-averse training.)

We will incorporate this discussion and explanation into the revised manuscript. Thank you again for your constructive suggestion.

---

[R3Q3] (Q3, W7) While the illustrations are very well done, the choice of competing algorithms is not very well justified.... / Can the authors explain how they chose the other methods they compared to?

[R3A3] Thank you for this question. In our results table, SGM uses the same configuration but without importance sampling. This isolates the contribution of our pretrained-model loss guided importance sampling scheme and demonstrates the benefit. We selected DORO and $\chi^2$-DRO to assess whether state-of-the-art robust optimization methods can address the tail-risk challenges without our proposed mechanism- i.e., importance samples. The results highlight that such methods do not inherently resolve the difficulties addressed by RAMIS, underscoring the need for our new approach. Finally, ERM is included as a conventional baseline to illustrate the limitations of average-risk optimization in high-risk scenarios.

We will clarify this motivation in the revised version.

---

[R3Q4] - (W4) The overall idea hinges upon the assumption that the generative model is sufficiently good to generate high-risk data points when such high-risk data points are not present in the data. It would be best to formalize this, i.e., what generalization capabilities are assumed on the generative model for the overall approach to work...

[R3A4] As the reviewer notes, our framework assumes that the generative model can sample from the true distribution $p(x)$, including high-risk regions. This is feasible under our setup as we use a score-based generative model, where the input to our importance sampler is the time-dependent score $\nabla \log p_t(x)$. This implies that the model can generate samples following $p(x)$.

We will formalize this assumption more clearly in the revised manuscript.

---

[R3Q5] (Q4, W3) CVAR is not the only risk-sensitive measure that could be used.... adequate estimation of tails are always at the core of distributionally robust learning even if some other risk measure than the CVAR is used / Can the authors try experiments using extremile rather than the CVAR? [R3A5] We appreciate the reviewer's suggestion. Our framework is indeed general and can be applied to other risk measures.

Following the Extremile formulation in [2] and the notations, we use: various $\tau$ values raning from 0.5 to 0.99, $r(\tau)=\log(1/2)/\log(\tau)$, $K_{\tau}(t) = t^{r(\tau)}$, and $J_{\tau}(F(x))$ which is scaled by the squared loss. We compare RAMIS + Extremile (our framework applied to Extremile with importance sampling) against naive Extremile optimization under the same experimental setup as in Figure 2 of the paper.

In all settings, RAMIS with importance sampling outperforms the naive baseline, confirming both the method’s flexibility and effectiveness across different risk measures. We will include these new results in the revised manuscript.

---

We appreciate the reviewer’s valuable feedback again. We believe we have addressed all concerns; if so, we kindly ask the reviewer to consider a higher score. We are happy to provide additional clarification if needed.

[R2Q1] - ...the effect that the method has on the standard risk. In Table 2 we start to see ERM improve over RAMIS for lower values of $\beta$. While its expected that methods that optimize the CVaR will not also optimize the standard empirical risk, it would be nice to show these results in the table as well --- even if just to understand the tradeoffs.

[R2A1] We thank the reviewer for the insightful comment.

In Table 2, each row corresponds to a different value of $\beta$. As $\beta$ decreases, the CVaR objective gradually approaches the standard ERM. In fact, when $\beta = 0$, CVaR becomes equivalent to ERM, and RAMIS can be directly used for minimizing the standard empirical risk. To fully address the reviewer’s suggestion, we conduct two complementary analyses.

(1) Standard ERM performance with $\beta = 0$: First, based on the same configuration corresponding to Table 2, we set $\beta = 0$ for the objective and all methods under the standard ERM. Below table shows the results (in unit of $10^{-3}$).

In this case, the CVaR objective function is equivalent to ERM. As they are minimizing the same objective function, their performance is nearly identical, as expected.

(2) Evaluating trained models at different non-target quantile levels:

Next, we fix $\beta = 0.99$ (1% tail-risk focus) and evaluate the resulting models across a range of quantile levels $\beta$

These results highlight the expected tradeoff: RAMIS, the proposed method, achieves superior performance in target high-risk regimes (target $\beta=0.99$ and $\beta=0.95, \beta=0.9$), while ERM excels in average-case scenarios. This supports the intuition that optimizing for CVaR enhances robustness to rare, high-loss inducing samples, at the potential cost of standard performance.

We will include these results and discussions in the revised manuscript. We thank the reviewer again for raising this important point.

---

[R2Q2] - The empirical results are a bit weak (as I believe the authors already somewhat acknowledged in their discussion of limitations and future directions)...

[R2A2] Our synthetic experiment was intentionally designed as a controlled setting to rigorously evaluate the core mechanism of RAMIS, whether our framework can identify and generate high-loss inducing samples that are often missed by standard sampling, by using a pretrained initial model. By constructing a Gaussian mixture with extreme mode imbalance (Fig. 2), we create a challenging yet interpretable testbed to directly visualize sample coverage and assess risk sensitivity.

We evaluate RAMIS on wireless CSI compression to demonstrate its applicability to high-dimensional, practical data where compression robustness to rare but high-distortion channel conditions is essential. This task exemplifies a real-world setting where standard training underrepresents such cases, and improved risk sensitivity directly translates to better signal reconstruction under challenging conditions.

That said, we agree that applying RAMIS to broader domains is an important next step, and we plan to explore such extensions in future work.

---

[R2Q3] - I understand that the fixed importance sampling distribution is computationally efficient, but am curious about how sensitive this is to the choice of the initial model. For example, if is uniformly bad, it will not be informative. Is there any other reason not to iterative update the reference model used for loss guidance?

[R2A3] Thank you for this question.

[Fixed Importance Sampling Distribution] As the reviewer noted, a primary motivation for using a fixed importance sampling distribution is computational efficiency: it avoids repeated loss evaluations, recomputation of the normalization constant, and allows reuse of generated samples. Iteratively updating the sampling distribution per optimization step introduces significant overhead, as highlighted in prior literature (e.g., Zhao & Zhang, 2015). Rather than performing per-step updates, our practical importance sampling distribution design focuses on minimizing the stochastic noise bound, i.e., the final expected model gap.

[Choice of $\theta_0$] Moreover, our framework is designed for the fine-tuning setting, assuming access to a reasonably informative pretrained model $\theta_0$, as reflected in the title of this work. Pretrained models for many tasks are now widely available and can serve as strong baselines and provide informative guidance for importance sampling. In typical fine-tuning scenarios, only a small number of additional training iterations are required compared to training from scratch. Empirically, we demonstrate that using a fixed importance sampling distribution in this regime yields superior performance compared to existing robust optimization baselines.

If a pretrained model $\theta_0$ assigns uniformly low or high losses, it indicates the absence of a meaningful loss tail—i.e., a scenario where risk-sensitive training is unnecessary.

Using an untrained or randomly initialized model (e.g., $\theta_0 = 0$) leads to uninformative importance weights, since the loss-induced distribution fails to highlight relevant failure modes. Such scenarios are outside the scope of our intended use case, which targets the increasingly common setting where pretrained models are available and serve as reasonable reference surrogates.

[Iterative Application] That said, RAMIS is fully compatible with iterative refinement. Specifically, Algorithm 1 can be applied recursively: the model $\theta_K$ obtained from one round can serve as the new reference model for generating the next round of guided samples. Our theoretical analysis remains valid under this substitution, and the bound in Eq. (7) becomes strictly tighter due to improved reference model quality.

In our experiments, the fixed pretrained-model loss guided importance sampling distributions already achieve strong CVaR performance. However, when starting from an untrained model that requires many iterations to converge, repeated application of RAMIS will lead to improved performance.

We will clarify these points in the final version of the manuscript and explicitly state that RAMIS supports iterative application when appropriate.

[Ref-R2-1] Zhao, Peilin, and Tong Zhang. "Stochastic optimization with importance sampling for regularized loss minimization." International Conference on machine learning. PMLR, 2015.

---

[R2Q4]- L256 / L291 / Table 1 acronym confusion

[R2A4] Thank you for the careful proofreading. In L256 and Table 1, "SGM" refers to the Stochastic Subgradient Method, where SPL is a typo, which should be SGM. We will revise the manuscript to remove redundant or conflicting abbreviations.

---

We appreciate the reviewer’s valuable feedback again. We believe the revised discussion addresses all the concerns, and we kindly ask the reviewer to consider a higher score. We are happy to provide additional clarification if needed.

We thank the reviewer for the constructive and thoughtful feedback. For clarity, we have merged related points from the review into grouped responses below.

[R1Q1] Convergence rate. The paper does not compare or comment on their theoretical analysis with past work... / Can any comparison be made about the theoretical convergence rate between the proposed method and other benchmark methods? ... rate presented in this paper compared to the rates SGM, DORO, etc.? / How does the analysis differ from previous works?

[R1A1] Thank you for the valuable comment. Under Assumption 1, our method achieves an $\mathcal{O}(1/\sqrt{K})$ convergence rate in iteration complexity, as shown in Theorem 1. This rate is consistent with the rate for the baseline objectives, though derived under distinct assumptions as part of them having distinct objective function. For instance, DORO [Zhai et al., 2021] achieves the same rate under the assumption that the corresponding objective (line 991) satisfies the same standard convexity and Lipschitz continuity (Assumption 1) and $\chi^2$-DRO can rely on a finite f-divergence constraint and employ mirror descent [Namkoong & Duchi, 2016], achieving the same rate. Optimization with the ERM objective also achieves $\mathcal{O}(1/\sqrt{K})$ without any changes in assumptions. SGM is the fair baseline, as it has the same CVaR objective. Notably, our method reduces to SGM when the importance weight $w^*$ is a constant (cf. Eq. (7)), yielding the same convergence rate $\mathcal{O}(1/\sqrt{K})$. This equivalence is expected, typically, importance sampling does not improve the asymptotic convergence rate, but rather the constant factor via noise reduction [Zhao & Zhang, 2015].

The key theoretical distinction of our work lies in reducing the stochastic noise floor via pretrained loss-guided sampling: We provide a bound on the stochastic noise term $\hat{v}(q)$, which is minimized when sampling from $q(x)\propto w^* (x)p(x)$ where $w^*(x)=(\sqrt{ 2L_{1}\ell(\theta_0;x)}+2L_{2} \kappa)^2 + 1$. Note that this analysis differs from previous works that use importance sampling, as we explicitly connect the importance sampling weight function to the loss value of the given pretrained model. This analysis enables a practical design of $\varphi(\ell)$, and under Assumption (9), the pretrained-loss guided sampling distribution achieves a tighter upper bound on the stochastic noise compared to standard sampling (e.g., SGM).

None of the existing baselines in CVaR optimization or importance sampling provides a stochastic-noise-sensitive analysis that incorporates pretrained models into the sampling distribution.

We will add this discussion and the distinct theoretical contribution in the revised manuscript.

---

[R1Q2] Empirical results that compare different choices of $\varphi$? / How to choose $\varphi$?

[R1A2] We thank the reviewer for raising this important question.

The desired form of $\varphi$ is derived in Eq (7) and given by $\varphi(\ell)=(\sqrt{\ell}+\frac{2L_{2}\kappa}{\sqrt{2L_{1}}})^2 + 1/(2L_1)$ as stated in the Appendix (line 1116). This form arises from matching the optimal importance sampling distribution that minimizes the stochastic noise $\hat{v}(q)$ in Thm 1.

However, in practice, the exact constants $L_{1}, L_{2}$ and $\kappa$ are typically unknown or difficult to estimate. To address this, we propose a practical surrogate $\varphi(\ell) = \ell + c$ where $c$ is a tunable constant. This approximation preserves the dominant order of the loss and offers a simple yet effective implementation.

To empirically study the effect of different $\varphi$, we conducted ablations in Appendix, Section E.2 (Impact of Importance Level Emphasis). We varied $c\in \{0.001, 0.01, 0.1, 1, 10\}$ and evaluated performance across a range of $\beta$ values (quantile). The results show that the method is robust to the choice of $c$, especially at high $\beta$, (e.g., for $\beta=0.99$ the proposed method consistently outperforms all the baselines with all $c$. For $\beta=0.5$, $c\in \{0.001, 0.01, 0.1\}$ lead to performance improvement). It indicates that when large-loss-inducing samples are strongly emphasized, the precise tuning of $\varphi$ becomes less critical.

We agree that this analysis is important and will add this discussion to the main paper in the revised submission for clarity.

---

We thank the reviewer again for the valuable feedback. We believe the revised discussion addresses the concern, and we respectfully ask the reviewer to consider increasing their score. We would be happy to provide further clarification if helpful.

[Ref-R1-1] Zhao, Peilin, and Tong Zhang. "Stochastic optimization with importance sampling for regularized loss minimization." International Conference on machine learning. PMLR, 2015.

Dear Reviewer,

Thank you again for your valuable and constructive feedback. We hope our detailed responses have addressed your concerns—particularly regarding the convergence rates and the empirical design of the importance weighting scheme. Please let us know if any further clarification or elaboration on these points would be helpful.

Best regards,

the authors