PRIME: Deep Imbalanced Regression with Proxies

We appreciate the reviewer’s feedback and the opportunity to clarify our claim regarding state-of-the-art (SOTA) performance. Below, we provide detailed comparisons with VIR and IM-Context, along with additional experiments to ensure a fair evaluation.

1. Comparison with VIR

Our claim of achieving SOTA performance is based on evaluations under a unified experimental protocol: using the same train/val/test splits, backbone (vanilla ResNet-50), and training settings as in prior works such as LDS, RankSim, and ConR. This setup allows for a fair and controlled comparison that isolates the contribution of the proposed PRIME framework.

In contrast, VIR uses a different data split and a more complex model architecture, incorporating a calibration network and reconstruction modules. These differences make direct comparisons of reported performance potentially misleading, as the results reflect not only the learning objective but also auxiliary components and dataset configurations.

In response to the reviewer’s suggestion, we conducted an additional experiment comparing PRIME to VIR under VIR’s official setting. Since the code and data splits for VIR are only publicly available for AgeDB-DIR, we focused our comparison on this dataset. We trained PRIME using the same train/val/test splits as VIR, while keeping all other settings unchanged. We then compared our results against the official pre-trained VIR model provided by the authors. For PRIME, results are averaged over five runs. As shown in Table 1, PRIME consistently outperforms VIR across all evaluation metrics under this setting.

We will revise the manuscript to include this experiment and clarify the scope of our SOTA claim accordingly.

Table 1. Comparison with VIR.

2. Comparison with IM-Context

We thank the reviewer for bringing IM-Context to our attention.

First, we note that IM-Context is built upon a substantially different model configuration. It employs a pre-trained CLIP image encoder to extract visual features, followed by in-context learning using large-scale models such as GPT-2 and PFN. In contrast, PRIME uses a ResNet-50 backbone trained from scratch, without leveraging any pre-trained models. Therefore, direct comparisons may not be meaningful, as the two approaches differ significantly in both model capacity (i.e., CLIP, GPT-2, PFN vs. ResNet-50) and training paradigm (i.e., leveraging pre-trained models vs. training from scratch).

Second, we emphasize that PRIME is a proxy-based representation learning framework designed to be independent of model architecture. Indeed, our method is broadly applicable and can benefit from stronger backbones. We expect that using more powerful models would further improve performance.

To examine this, we conducted additional experiments using the pre-trained CLIP image encoder (ViT-B/32) as the backbone, which is also used in IM-Context. Specifically, we fine-tuned the CLIP backbone together with a two-layer MLP regression head and trained the model using the PRIME loss. To ensure robustness, we report the average performance of PRIME over five independent runs. We then compared our results on AgeDB-DIR and IMDB-WIKI-DIR with those reported by IM-Context.

As shown in Tables 2 and 3, PRIME substantially outperforms both PFN-localized and GPT2-localized across all evaluation metrics. These findings confirm that PRIME remains effective even on top of strong pre-trained models, demonstrating its flexibility across backbone choices. We believe these additional experiments further support our SOTA claim under a unified and fair evaluation protocol.

We will revise the manuscript to include these results and reflect the comparison with IM-Context.

Table 2. Results for AgeDB-DIR.

Table 3. Results for IMDB-WIKI-DIR.

We appreciate the reviewer’s comments. We have addressed all points and will revise the manuscript accordingly.

1. Impact of hyperparameters

We have already provided detailed analyses of the impact of each hyperparameter ($\lambda_p$, $\lambda_a$, $\tau_f$, $\tau_t$, and $\alpha$) in Tables 16–20 in the appendix. The results show that PRIME consistently outperforms the w/o PRIME baseline regardless of the choice of hyperparameters, demonstrating strong robustness and practical stability. We will revise the manuscript to make this analysis more prominent.

2. Clarification on proxy initialization

We believe there may be a misunderstanding. PRIME does not rely on any optimal or specific initial placements of proxies. As stated on the right side of line 115 in the manuscript, all proxies are randomly initialized and jointly learned during training. Moreover, the effect of initialization randomness is already reflected in the repeated experiments with different random seeds, as reported in our main tables. The consistent performance across these runs confirms that PRIME is robust to proxy initialization. We will revise the manuscript to make this clearer and avoid potential confusion.

3. Theoretical analysis under non-optimal proxy positioning

We note that the derivation of the generalization error bound in Theorem 4.1 remains valid regardless of whether the proxies are optimal. When proxies are not optimally positioned, the resulting discrepancy can be incorporated as an additional term in the bound, rather than invalidating the analysis. To support this, we provide a sketch of how the generalization bound can be extended to the non-optimal case.

Let $\tilde{\mathbf{z}}_j^p$ for $j=1,\ldots,C$ denote the optimal proxy features,

and $\tilde{p}:=\tilde{p}_{\theta}(\xi|\mathbf{x})$ be the corresponding feature association distribution.

The empirical (i.e., non-optimal) proxies can be defined as $\mathbf{z}_j^p:=\tilde{\mathbf{z}}_j^p+\epsilon_j$,

where $\epsilon_j$ represents the estimation error, and $p:=p_{\theta}(\xi|\mathbf{x})$ denotes the corresponding feature association.

We revisit the balanced alignment risk term in Eq. (21) of the appendix, which was originally formulated based on optimally positioned proxies. To analyze the non-optimal case, we subsititue $\log \tilde{p}$ by applying the following identity: $\log \tilde{p}=\log p + [\log \tilde{p} - \log p].$

Then, the first term, $\log p$, follows the same derivation as in Theorem 4.1. The second term, $\log\tilde{p}-\log p$, captures the discrepancy introduced by the deviation between the empirical and optimal proxies. Since $p$ and $\tilde{p}$ is defined as softmax, we can bound this residual term using the inequality: $\frac{\sum a_i}{\sum b_i} \leq \max_i \frac{a_i}{b_i}$ for positive values $a_i$, $b_i$, which leads to: $|\log\tilde{p} - \log p| \leq 2\tau_f\max_j |d_f(\mathbf{z},\tilde{\mathbf{z}}_j^p + \epsilon_j) - d_f(\mathbf{z},\tilde{\mathbf{z}}_j^p)|.$

Finally, applying the inequality above to Eq. (21) yields an additional bounded term, which can be directly incorporated into the generalization error bound derived in Theorem 4.1. Importantly, as training progresses and the proxies become more accurate (i.e., $\epsilon_j$ becomes smaller), the residual decreases accordingly, leading to a tighter bound. Our empirical results also show that PRIME performs robustly with random initialization of proxies. We will include this extended derivation in the revised manuscript with a full proof.

4. Distinction from classification-based methods

To the best of our knowledge, PRIME is the first to introduce proxies specifically designed for imbalanced regression. Moreover, as discussed in Section C.4 of the appendix, PRIME differs substantially from prior classification-based regression methods in its formulation and learning objective.

First, existing approaches typically quantize continuous targets into discrete bins and treat each bin as a class, which inevitably introduces quantization error. In contrast, PRIME assigns proxies based on soft associations (Eq. (6)) derived from pairwise target distances, effectively mitigating such errors.

Second, instead of predicting proxy indices (i.e., classes), PRIME learns to minimize the distance between features and their associated proxies in the embedding space. This objective aligns more naturally with the continuous nature of regression and promotes representations that reflect target similarities.

Furthermore, we also empirically compare PRIME with the most recent classification-based method, Hierarchical Classification Adjustment (HCA) [CVPR’24], and observe consistently superior performance, as shown in Tables 1–3 of the manuscript. We will revise the main text to better highlight these distinctions and clarify the novelty of our method.

5. Related work

As the comment does not specify which references are missing, we would appreciate any suggestions the reviewer might have.

We thank the reviewer for the constructive and insightful comments. We have addressed all points and will revise the manuscript accordingly.

1. Ablation on non-learnable proxy

As noted by the reviewer, PRIME is flexible with respect to how proxies are constructed—proxies can be either learnable or non-learnable. Following the suggestion, we conducted an additional ablation experiment on a non-learnable variant of PRIME, where proxy features are updated as the centroids of sample features assigned to each proxy. Specifically, to ensure the proxy loss and alignment loss can be properly backpropagated, we update the proxy features within each mini-batch based on the current sample-to-proxy assignments.

Table 1 compares the performance of the centroid-based proxy update with our PRIME using learnable proxies on the AgeDB-DIR dataset. Results are averaged over five runs. While the centroid-based method achieves slightly better performance than the learnable proxy in the Median category, it suffers from notable performance degradation in the other regions. In particular, we observe a significant performance drop in the Few category, indicating that the centroid-based proxies struggle under severe data sparsity.

We attribute this performance gap to the inherent limitations of the centroid-based method. Since proxies are updated as the centroids of the assigned sample features, their quality heavily depends on the number of assigned samples. When only a few samples are available—as is often the case in the Few category—the estimated centroids become unstable and unreliable. Moreover, because centroids are computed within each mini-batch, their estimates can fluctuate significantly depending on the batch composition. In contrast, learnable proxies are global parameters that are updated via backpropagation, offering greater stability and robustness, particularly in data-sparse regions.

This issue is further exacerbated in regression settings, where some proxy bins may contain no samples at all. In such cases, the centroid-based approach cannot update the corresponding proxies, leaving them inactive throughout training. In contrast, learnable proxies remain effective even without sample assignments, serving as global reference points that guide other samples and support representation learning.

We will add this ablation study and discussion to the revised manuscript.

Table 1. Results with non-learnable proxies.

2. Clarification on Figure 2(b)

The twisted line pattern in Figure 2(b) appears due to suboptimal alignment between features and their corresponding proxies in the Few category. Although the proxies represent a balanced feature distribution, the alignment process in Eq. (7) still faces challenges under sample imbalance. Minority samples, which occur infrequently, often fail to align properly with their proxies, resulting in distorted feature–proxy alignment.

The use of class imbalance techniques (e.g., PRW, CB, and LDAM) provides better alignment focus on minority samples, mitigating this issue. To empirically validate their effect, we conducted an additional analysis on the AgeDB-DIR dataset, measuring the Spearman correlation between the proxy–feature similarity matrix (as visualized in Figure 2(b)) and the label similarity matrix. A higher correlation indicates better alignment and reduced distortion in the learned feature space.

Table 2 reports the Spearman correlation values when PRIME is combined with various class imbalance techniques. Results are averaged over five runs. Incorporating class imbalance techniques significantly improves the correlation, confirming their effectiveness in facilitating better alignment, particularly for samples in the Few category.

We will revise the manuscript to include this additional analysis and the corresponding figures alongside Figure 2(b).

Table 2. Alignment between proxies and features.

We thank the reviewer for the constructive feedback. We have addressed all points and will revise the manuscript accordingly.

1. PRIME’s effectiveness in the Few category

We would like to point out that PRIME alone achieves state-of-the-art performance in the Few category on three out of four datasets, outperforming existing methods by 0.7%, 0.5%, and 3.6% on AgeDB-DIR, NYUD2-DIR, and STS-B-DIR, respectively. Furthermore, a key contribution of PRIME lies in its ability to seamlessly integrate class imbalance techniques into regression tasks. This facilitates balanced feature learning, substantially enhancing performance in the Few category.

To further validate PRIME’s effectiveness in the Few category, we conducted an additional analysis on the test set of AgeDB-DIR, measuring the Spearman correlation between the feature similarity matrix and the label similarity matrix. A higher correlation indicates that the learned features are more well-ordered and better reflect the continuity of the label space, which is crucial for learning effective representations in regression tasks.

As shown in Table 1, we compared the Spearman correlations for all samples and for samples in the Few category across PRIME, RankSim, and ConR. Results are averaged over five runs. While RankSim and ConR exhibit reasonable correlation values on the full dataset, their performance significantly degrades in the Few category. In contrast, PRIME maintains a high correlation within the Few category, indicating that it learns well-ordered feature representations, even for minority samples. This highlights the strength of our proxy-based formulation, which provides holistic guidance for feature positioning, allowing minority samples to be embedded in alignment with the overall label structure.

We will incorporate these additional results and analyses into the revised manuscript to better highlight PRIME’s effectiveness in the Few category.

Table 1. Spearman correlation between feature and label similarities.

2. Feature space uniformity

The second term in Eq. (4) encourages proxies to be repelled from one another, as it increases the angle between $\mathbf{z}_i^p$ and $\mathbf{z}_j^p$, thereby inducing a more dispersed (i.e., uniform) proxy distribution. Unlike the original uniformity loss (Wang & Isola, 2020), which pushes all pairs equally apart, our formulation increases the pairwise cosine distance proportionally to the label distance. This not only promotes uniformity in the proxy space but also preserves label ordinality.

Since sample features are aligned to proxies via the alignment loss in Eq. (7), the proxy uniformity is naturally transferred to the feature distribution. In this way, the second term in Eq. (4) promotes feature space uniformity by shaping the proxy distribution, which in turn guides the feature distribution through alignment.

To validate this, we trained a variant of PRIME without the second term in Eq. (4) (i.e., setting $\alpha=0$) and measured feature space uniformity on the test set of AgeDB-DIR using the $\mathcal{L}_{\text{uniform}}$ metric proposed by (Wang & Isola, 2020).

Table 2 presents the results averaged over five runs. Removing the second term leads to higher $\mathcal{L}_{\text{uniform}}$ (i.e., lower uniformity), confirming that this term is critical for inducing feature space uniformity. We will include this result and discussion in the revised manuscript.

Table 2. Feature space uniformity.

3. Larger number of proxies

For simplicity, we set the number of proxies $C$ such that each proxy would roughly cover a 5-year age span for AgeDB-DIR and IMDB-WIKI-DIR, and kept it fixed while tuning the other hyperparameters. As shown in Table 15 in the appendix, even without tuning, increasing the number of proxies leads to improved performance in the many and median regions. This suggests that with proper hyperparameter tuning, a larger number of proxies could further improve performance.

To support this, we conducted an additional experiment with $C = 40$, tuning the associated hyperparameters ($\lambda_p = 1$, $\lambda_a = 5$, $\tau_f = 5$, $\tau_t = 1$, $\alpha = 0.005$), and obtained even better results, as reported in Table 3. Results are averaged over five runs. We will revise the manuscript to include this result.

Table 3. Results with more proxies on AgeDB-DIR.