Variational Supervised Contrastive Learning

We thank you for your feedback and for recognizing our confidence-adaptive temperature as an interesting idea. We appreciate this opportunity to address potential misunderstandings regarding our empirical results and theoretical contributions. We hope to clarify VarCon's effectiveness by addressing each of your concerns below:

Regarding VarCon's performance gains (W1)

We respectfully disagree with the claim that our improvements are "within standard error." This appears to be a factual misunderstanding of our results.

Table 1: Statistical significance of VarCon's improvements over SupCon. All Top-1 accuracy values are directly from Table 1 in our main paper. We add statistical significance analysis showing the number of standard errors ($\sigma$) calculated as: $\sigma = \frac{\text{Gain}}{\sqrt{\text{SE}\_{\text{VarCon}}^2 + \text{SE}\_{\text{SupCon}}^2}}$, where SE denotes standard error. Even our smallest improvement ($4.7\sigma$) is highly significant.

All improvements over SOTA SupCon are highly statistically significant, ranging from $4.7\sigma$ to $13.4\sigma$. Additionally, we evaluate few-shot learning performance where $N$ denotes the number of training samples per class from ImageNet's 1000 classes. For each setting, we randomly sample $N$ images per class for training (if a class has fewer than $N$ samples, all samples are retained). With $N=50$, both methods must learn to distinguish 1000 classes using merely 50,000 total samples, which is an extremely challenging scenario resulting in ~2.5% accuracy for both approaches. As data increases to N=100, 200, and 500, VarCon demonstrates clear advantages over SupCon (37.81% vs 36.57%, 51.10% vs 50.25%, and 65.83% vs 64.91% respectively), with improvements ranging from $3.3\sigma$ to $4.2\sigma$ significance. At $N=1000$, both methods reach similar performance (~73%) as they approach the ceiling achievable with this limited per-class data. Importantly, when trained on the full ImageNet dataset where no such ceiling exists, VarCon's advantage expands to $8.1\sigma$, demonstrating that our variational framework's benefits scale with data availability. We faithfully report all results across the entire data spectrum to provide a complete picture of our method's performance characteristics.

Significance of ELBO derivation (our core contribution) (W2, Q1)

We respectfully disagree with the characterization of ELBO derivations and the number of training epochs as "less advanced areas" and "not central to main contribution". We believe these comments stem from a major misapprehension of our technical contribution and would like to further elaborate on that. Firstly, Section 3.1's derivation of the ELBO of Class-Conditional Likelihood is the key theoretical foundation of our approach and is a novel and non-trivial contribution - establishing connection between contrastive learning and variational inference for the first time. This derivation lay the foundation of the formulation of our VarCon loss that explicitly regulates embedding distributions and enforces appropriate semantic relationships between samples. It also motivate our confidence-adaptive temperature mechanism, which addresses the fundamental challenge of hard samples being difficult to learn while easy samples overfit. Moving this to the appendix would obscure our core theoretical contribution. We were encouraged that other reviewers (gzPQ, s4KC, 5QVM) unanimously recognized the significance of this theoretical contribution. To prevent any future misinterpretation, we will revise the manuscript to more explicitly highlight the centrality and novelty of this framework from the outset.

Secondly, we argue that interperting an established subject from a novel variational angle is fundamental to modern deep learning and comprises enough novelty and significance, as evidenced by:

[1] Kingma, Diederik P., and Max Welling. Auto-Encoding Variational Bayes. International Conference on Learning Representations (2014).

[2] Diederik P. Kingma, and Ruiqi Gao. Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation. Advances in Neural Information Processing Systems 36 (2023).

Regarding the importance of training epoch analysis, this analysis directly validates our method's efficiency. While SimCLR requires 1000 epochs, MoCo needs 200-800 epochs, and SupCon needs 350 epochs for peak performance, VarCon achieves superior results in just 200 epochs: a 1.75-5× speedup with significant practical implications for computational resources and research accessibility.

Regarding content placement in main paper, the ELBO derivations must remain in the main text as they constitute our core theoretical contribution detailed above. This derivation directly yields our loss formulation and confidence-adaptive temperature mechanism. Relocating this material to the appendix would obscure the theoretical foundation of our work. Similarly, ablation studies warrant inclusion in the main text as they demonstrate that our improvements arise from principled design choices rather than parameter tuning, establishing robustness and reproducibility—essential criteria for methodological contributions.

Comprehensiveness of our experimental verification (W3, Q2-4)

Our empirical evaluation comprehensively demonstrates VarCon's advantages: Section 4.1 presents SOTA classification results on 4 datasets with 4 architectures; Section 4.2 shows consistent improvements across all few-shot learning regimes; Section 4.3 validates superior transfer learning on 12 diverse tasks; Section 4.4 tests robustness across different augmentations and architectures; Section 4.5 provides comprehensive ablation studies; Section 4.6 demonstrates superior corruption robustness on ImageNet-C; Appendix B.3 visualizes clear semantic organization evolution; and Appendix B.4 quantifies representation quality through clustering metrics. To our best knowledge, our paper provides one of the most comprehensive experimental validations in recent contrastive learning literature. We would appreciate specific suggestions on what additional demonstrations would be needed should this reviewer found the current results insufficient.

Regarding baseline selection criteria (Q2)

We selected models based on both criteria. Table 1 includes the most established SOTA methods in single-modality contrastive learning: SimCLR, MoCo V2, BYOL, SwAV, VicReg, and Barlow Twins. For supervised methods, we included Cross-Entropy (standard baseline) and SupCon (current SOTA). We tested across ResNet-50/101/200 and ViT-Base, the most common architectures in visual contrastive learning. VarCon outperforms all baselines across every architecture, demonstrating our improvements stem from the method, not architectural advantages. We focus on single-modality (where paired multi-modal data is often unavailable) as it represents the fundamental challenge. Establishing superiority here is essential before extending to multi-modal settings.

Regarding comparison with importance weighting (Q3)

While specific importance-weighted approaches were not mentioned, we can clarify the fundamental differences from our method. Importance weighting scales loss values ($w_i \cdot L_i$) while keeping targets fixed. Our confidence-adaptive temperature $\tau_2(z) = (\tau_1 - \epsilon) + 2\epsilon p_\theta(r|z)$ instead modifies the target distribution itself, creating sharper distributions for hard samples (low $p_\theta(r|z)$) and smoother ones for confident samples (high $p_\theta(r|z)$). This represents a fundamentally different principle: dynamically shaping learning objectives rather than merely reweighting losses. If the reviewer has specific alternative methods in mind, we would be happy to discuss comparisons.

Regarding representation analysis (Q4)

Our paper provides extensive representation analysis across multiple sections. In Section 4.5, KNN classification (Figure 3a) demonstrates VarCon's superior class boundary separation through higher accuracy. Appendix B.3's t-SNE visualization (Figure 4) reveals clearer semantic clusters with reduced overlap compared to SupCon, while Appendix B.4's hierarchical clustering analysis (Table 6) quantifies this improved organization through enhanced ARI/NMI/purity metrics. These complementary analyses comprehensively validate VarCon's superior representation quality.

We deeply appreciate your high recognition our work. We thank you for highlighting our theoretical contributions and clear presentation. The rigorous probabilistic foundation enables a principled understanding of contrastive learning through variational inference. Our ELBO-derived objective with confidence-adaptive temperature enables highly efficient and robust training. We now address your main concerns below:

X-CLR empirical comparison (W1)

We acknowledge that we mentioned X-CLR in our paper (lines 71-73) as the most recent advancement in contrastive learning: "methods like X-CLR [48] further improve these frameworks by analyzing optimization dynamics and introducing graded similarities." While both methods utilize label supervision, they leverage labels in fundamentally different ways: VarCon uses labels purely for supervised learning (identifying which samples belong to the same class), whereas X-CLR extracts semantic information from class names to construct similarity relationships between different classes. This leads to several key differences that make direct empirical comparison challenging:

Key Methodological Differences:

VarCon and X-CLR address complementary scenarios: X-CLR leverages auxiliary text when available, while VarCon operates on visual data alone. As multi-modal paradigms become prevalent, VarCon provides a novel theoretical framework for understanding contrastive learning through variational inference. While naturally extensible to multi-modal settings, we first establish effectiveness in single-modality scenarios. Therefore, our comparisons focus on single-modality SOTA baselines (SupCon, SimCLR, etc.) to validate theoretical contributions before exploring multi-modal extensions.

Despite extensive efforts to locate X-CLR's implementation through official repositories and author pages, the code remains unavailable. Reproducing their method without original implementation risks introducing errors leading to unfair comparisons. Thus, we provide indirect comparisons using their reported numbers. We will clarify these methodological differences in our related work section and present detailed comparisons below.

Empirical Comparison with X-CLR through additional experiments (Q1)

Since X-CLR's code is not publicly available, we address the reviewer's concern by comparing against their reported results while closely matching their experimental settings. X-CLR trains for 100 epochs with batch size 1024 on ImageNet, using AutoAugment and a ResNet-50 encoder. We focus our comparison on ImageNet as it provides the most controlled setting: both methods use the same visual data and class labels, with the key difference being that X-CLR additionally leverages text-based similarities computed from class name descriptions using Sentence Transformer while VarCon operates solely on visual representations.

To ensure fair comparison, we trained VarCon with ResNet-50 (matching X-CLR) under various settings and present comprehensive results:

This allows direct comparison under identical conditions (row 2) while demonstrating VarCon's performance across different training configurations. Note that learning rates follow the linear scaling rule (lr $\propto$ batch_size/256), see main paper line 167. Even under X-CLR's exact batch size, VarCon outperforms by 1.65% on ImageNet and 0.8% on ImageNet Real, with the gaps increasing to 3.76% and 2.62% respectively under our full training regime.

Computational Requirements Comparison:

*Peak GPU memory with ResNet-50, batch size 2048. X-CLR should have similar GPU memory usage as SupCon during training since X-CLR pre-computes text similarities offline. In the table, $B$ denotes the per-device batch size and $C$ denotes the number of unique classes present in that batch.

Key Computational Advantages of VarCon:

We conducted detailed memory profiling experiments (to be included in the final manuscript) measuring peak GPU memory consumption over 50 training epochs on ImageNet with ResNet-50 and per-device batch size 512. VarCon demonstrates consistent memory efficiency advantages: VarCon maintains a steady peak memory usage of 43.09 GB per GPU compared to SupCon's 43.25 GB. While this difference appears modest, it becomes significant in multi-GPU training scenarios where even small per-device savings can enable larger effective batch sizes or accommodate additional model capacity within fixed memory budgets.

The memory savings stem from VarCon's architectural efficiency: by computing class centroids on-the-fly rather than exhaustive pairwise comparisons, VarCon reduces the memory required for storing intermediate similarity matrices from $O(B^2)$ to $O(B \cdot C)$. Crucially, $C$ (the number of unique classes per device) is bounded by $B$ (the per-device batch size), with $C \leq B$ always holding true.

Self-Adaptive Confidence-Based Temperature vs X-CLR's Approach:

While X-CLR leverages external text metadata to organize the embedding space through pre-computed semantic similarities between class descriptions, VarCon learns well-organized representations using only visual data through our ELBO-derived loss and confidence-adaptive temperature mechanism: $\tau_2(z) = (\tau_1 - \epsilon) + 2\epsilon p_\theta(r|z)$. This dynamically adjusts supervision strength, tightening for difficult samples and relaxing for confident ones, addressing the same challenge X-CLR tackles (avoiding SupCon's rigid binary similarities) but without requiring additional modalities.

As detailed in our response to Reviewer s4KC's Q1 (Table 1 on $\epsilon$ evolution), extensive experiments demonstrate this mechanism produces high-quality representations, including clustering quality (Section 4.5, Appendix B.4) and embedding visualization (Appendix B.3) on top of the existing experiments in the main body of our paper.

VarCon Learned Semantic Similarities (Key Examples)

VarCon successfully learns semantic relationships using only visual information: similar objects (felines: 0.43-0.50, sports balls: 0.50), demonstrating that our ELBO-derived objective organizes embeddings meaningfully without text-based similarities. We greatly appreciate X-CLR's pioneering work on leveraging multi-modal information to regularize embedding spaces, addressing crucial interpretability challenges in contrastive learning. While space constraints prevent replicating their full visualization (Figure 4), our results indicate that VarCon and X-CLR are complementary: X-CLR shows how auxiliary modalities guide representation learning, while VarCon achieves similar semantic organization through principled probabilistic frameworks alone. We will ensure X-CLR's contributions are properly highlighted in our revised Related Work section.

Minor issues

Thank you for noting the table ordering issue. We have corrected this in our revision.

We sincerely thank you for recognizing VarCon as "theoretically grounded and intuitively sound" with our ELBO derivation leading to "an intuitive combination of KL divergence and negative log-likelihood terms." We are particularly encouraged by your acknowledgment that VarCon "consistently outperforms both existing self-supervised and supervised contrastive learning baselines", achieving 79.36% on ImageNet (vs. 78.72% SupCon) and 78.29% on CIFAR-100 (vs. 76.57%), while converging in 200 epochs (vs. 350) with smaller batch sizes (2048 vs. 4096). Our confidence-adaptive temperature enables sample-specific learning for both hard samples and confident ones. We will address your comments as follows.

Regarding supervised framework (W1)

We respectfully note that supervised learning remains an important class of problems with abundant real-world applications. Our variational framework naturally extends to self-supervised settings by replacing class centroids $w_r$ with instance centroids $w_i$ computed from augmented views (similar to SimCLR's $z_i, z_j$ but using centroids instead of pairs). The confidence-adaptive temperature $\tau_2(z)$ would adapt based on the model's confidence in the instance centroid representation, analogous to how it currently adapts to classification confidence. We chose supervised settings first to establish theoretical soundness, with self-supervised extensions as promising future work.

Additional adaptive temperature analysis (W2)

We appreciate your interest in our confidence-adaptive temperature mechanism. Section 4.5 and Figure 3b show that $\epsilon=0.02$ yields optimal ImageNet performance, while even $\epsilon=0$ (no adaptation, ELBO-only) achieves 78.87%, surpassing SupCon. This demonstrates VarCon's effectiveness stems from both ELBO-derived loss and adaptive mechanism. Since $\epsilon$ is learnable, we provide comprehensive experiments tracking $\epsilon$ evolution and sample-specific $\tau_2$ distributions throughout training below.

Temperature evolution analysis (Q1)

We have provided comprehensive experimental analysis demonstrating our adaptive temperature mechanism's effectiveness throughout the paper. This includes superior performance across all benchmarks (Sections 4.1-4.4, 4.6; Appendix B.2), systematic ablation of $\epsilon$'s impact (Section 4.5, Figure 3b), and quantitative embedding space evaluation via KNN classification and hierarchical clustering (Appendix B.3-B.4). To further address your insightful questions, we now provide detailed tracking of $\epsilon$ evolution and sample-specific $\tau_2$ distributions throughout training. We address each of your three sub-questions below.

Evolution of learnable $\epsilon$

Table 1: Distribution of adaptive temperature $\tau_2$ across ImageNet samples at key epochs during 200-epoch training with ResNet-50, $\tau_1 = 0.1$, and batch size 4096. The temperature $\tau_2$ modulates gradient strength, with lower values indicating stronger gradients for difficult samples.

Learnable $\epsilon$ increases from 0.0185 (epoch 40) to peak at 0.0215 (epoch 120), then stabilizes at 0.0202 (epoch 200). Mean $\tau_2$ rises 13.6% (0.09365→0.10642), reflecting growing model confidence. Key distributional shifts: 25th percentile +17.3% (0.08542→0.10028), median +19.9% (0.09205→0.11030), 75th percentile +14.0% (0.10098→0.11508), indicating progressively confident predictions and gentler gradients. Minimum $\tau_2$ remains ~0.08 (persistent hard samples), maximum ~0.119. Standard deviation peaks at 0.01088 (epoch 160) before decreasing to 0.01056 (epoch 200).

This evolution enables distinctive feature learning early via stronger gradients on uncertain samples, while later stabilization prevents drift. Persistent low $\tau_2$ ensures hard samples continue improving; higher $\tau_2$ maintains stable clusters for confident samples. This adaptive mechanism explains VarCon's superiority over fixed $\epsilon=0$ (79.36% vs. 78.87% on ImageNet).

Intra-class dispersion improvement

Yes, our adaptive temperature mechanism demonstrably improves intra-class dispersion through multiple quantitative measures. We provide three forms of empirical evidence:

Visual analysis: Appendix Section B.3 shows VarCon achieves 79.11% KNN accuracy after 200 epochs versus SupCon's 78.53% after 350 epochs, demonstrating both faster convergence and superior representations. The largest gains occur during epochs 100-150 (59.57% to 71.12%) when the KL divergence term aligns posteriors. This efficiency stems from: class-centroid alignment, probabilistic inter-class relationships, and confidence-adaptive supervision.

Quantitative analysis: We evaluate embedding quality through two complementary approaches. First, KNN classification (Section 4.5, Figure 3a) directly probes representation quality without learnable parameters that could mask poor features. VarCon consistently outperforms SupCon across all K values, demonstrating superior local neighborhood structure. Second, while Appendix Section B.4 and Table 6 already present hierarchical clustering results, we appreciate your suggestion to ablate the adaptive temperature. Table 2 includes additional results with VarCon fixed $\epsilon$=0 (constant $\tau_2=\tau_1=0.1$):

Table 2: Hierarchical clustering evaluation on ImageNet embeddings using Ward linkage. All metrics: higher is better.

These metrics evaluate clustering quality: ARI measures agreement with ground truth correcting for chance; NMI quantifies mutual information between clusters and labels; Homogeneity ensures each cluster contains only one class; Completeness ensures each class is in one cluster; V-measure balances both; Purity measures fraction of correctly clustered samples. Progressive improvements from SupCon to VarCon ($\epsilon$=0) to VarCon demonstrate: (1) ELBO-derived loss alone improves semantic organization even without adaptation, and (2) adaptive temperature mechanism provides additional gains, validating both components' contributions to superior embedding structure.

Comparison with other regularization methods

We appreciate this insightful question. The adaptive temperature mechanism is specifically designed for our VarCon's loss—the KL divergence and confidence-adaptive temperature work synergistically in ways standalone regularization methods cannot replicate. Table 3 compares our approach with established techniques:

Table 3: Comparison of regularization methods (Dynamic Target: modifies target distribution based on sample/model state; Dynamic Gradient: adjusts gradient magnitude during training; Sample-Specific: per-sample treatment; Self-Adaptive: confidence-based updates without manual scheduling).

VarCon uniquely provides bidirectional adaptation: $q_\phi(r'|z)$ and $\tau_2(z)$ adjust simultaneously based on confidence: hard samples receive sharper targets with stronger gradients, easy samples get smoother targets with gentler gradients. Existing methods modify only one aspect. While optimized for VarCon, our adaptive temperature has broader implications for temperature-based methods including LLM training and generative models.

Typo correction (Q2)

Thank you for pointing out this typo: It should be $\log p_\theta(z)$ rather than $p_\theta(z)$. We have corrected this in our revised manuscript.

[1] Szegedy, Christian, et al. Rethinking the Inception Architecture for Computer Vision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016): 2818-2826.

[2] Zhang, Hongyi, et al. mixup: Beyond Empirical Risk Minimization. International Conference on Learning Representations (2018).

[3] Ghose, Arindam, and Balaraman Ravindran. AdaFocal: Calibration-aware Adaptive Focal Loss. Advances in Neural Information Processing Systems 35 (2022): 19998-20009.

[4] Tao, Linwei, et al. Dual Focal Loss for Calibration. International Conference on Machine Learning. PMLR, 2023: 33645-33660.

[5] Bahri, Dara, and Heinrich Jiang. Locally Adaptive Label Smoothing Improves Predictive Churn. International Conference on Machine Learning. PMLR, 2021: 532-541.

[6] Liu, Can, et al. Learning from Teaching Regularization: Generalizable Correlations Should be Easy to Imitate. Advances in Neural Information Processing Systems 37 (2024).

We greatly thank the reviewer for recognizing our theoretical grounding in variational inference, the consistent empirical gains across diverse tasks, and the architecture-agnostic nature of our method. We appreciate your constructive feedback and will address each concern below with additional ablations, visualizations, and comparisons to further validate our contributions.

Component ablations (W1/Q1)

We acknowledge the reviewer's request for component-wise ablations. We note that according to our ELBO derivation in Eq. (5):

$\mathcal{L}\_{ELBO} = D\_{KL}(q\_\phi(r'|z) || p\_\theta(r'|z)) + \log p(r) - \log p\_\theta(r|z) - \log p\_\theta(z)$

As discussed in lines 131-135, $\log p_\theta(z)$ would encourage high likelihood throughout the embedding space without class distinction, while $\log p(r)$ is a fixed constant based on the dataset's class distribution, which are are uninformative for contrastive learning. Therefore, our VarCon loss (Eq. 7) focuses on the contrastive-relevant components:

$\mathcal{L}\_{VarCon} = D\_{KL}(q_\phi(r'|z) || p\_\theta(r'|z)) - \log p\_\theta(r|z)$

Both terms are mathematically necessary to maintain the validity of our variational bound, removing either would break the theoretical connection to the ELBO. In Appendix Section C, we provide comprehensive gradient analysis demonstrating how these components affect learning dynamics. Specifically, we show the gradient flow w.r.t. embeddings $z$ and analyze how the sample-specific confidence-adaptive temperature $\tau_2(z) = (\tau_1 - \epsilon) + 2\epsilon p_\theta(r|z)$ modulates learning, where $\tau_1$ is fixed and $\epsilon$ is learnable.

Although both components are essential to the VarCon loss, we provide ablation results to isolate their individual contributions:

Table 1: Component ablation study on ImageNet with ResNet-50. All experiments use batch size 4096, 350 epochs, learning rate 2.0, and AutoAugment.

As noted in Section 4.5 (Figure 3b), even with $\epsilon=0$ (no adaptive temperature), VarCon achieves 78.87% Top-1 accuracy, already surpassing SupCon's 78.72%. The full model with adaptive temperature further improves performance, demonstrating the value of our confidence-based modulation mechanism.

Empirical evidence of superior representation quality (W2/W5/Q2/Q3)

We thank the reviewer for requesting empirical evidence on representation quality. We summarize existing results and provide additional experiments below.

Intra-class compactness: Figure 3(a) demonstrates VarCon's consistently superior KNN classifier performance across all K values, which directly reflect better local neighborhood structure. In Appendix B.3, Figures 4-5 show t-SNE visualizations where VarCon achieves clearer cluster separation after 200 epochs than SupCon after 350 epochs, evidenced by more distinct cluster boundaries and reduced inter-class overlap. Additionally, Appendix B.4 (Table 6) provides hierarchical clustering analysis, which inspects embedding quality without learnable parameters that could mask poor features:

Table 2: Hierarchical clustering evaluation on ImageNet embeddings using Ward linkage. All metrics: higher is better.

These metrics quantify clustering quality: ARI measures agreement with ground truth, NMI captures mutual information between clusters and labels, Homogeneity/Completeness ensure class-cluster alignment, V-measure balances both, and Purity measures correct assignments. VarCon's improvements across all metrics confirm superior semantic organization.

Difficulty-aware learning: We conducted comprehensive tracking of confidence-adaptive temperature $\tau_2$ throughout ImageNet training (200 epochs). Due to space constraints, we refer to our detailed response to Reviewer s4KC's Q1, which includes: (1) evolution of learnable $\epsilon$ from 0.0185 (epoch 40) to peak at 0.0215 (epoch 120) then stabilize at 0.0202 (epoch 200), (2) mean $\tau_2$ increasing 13.6% (0.09365 to 0.10642), (3) distributional analysis showing persistent low $\tau_2$ for hard samples (minimum stays ~0.08) while confident samples progressively receive higher $\tau_2$ (75th percentile rises from 0.10098 to 0.11508), confirming our mechanism adapts to sample difficulty as intended.

Comparisons with PaCo/GPaCo (W4)

We thank the reviewer for suggesting comparisons with PaCo and GPaCo. We note that these methods address orthogonal problems and could be combined synergistically:

Table 3: Comparisons with SupCon, VarCon, PaCo, and GPaCo.

VarCon reformulates supervised contrastive learning through variational inference to address SupCon's limitations (inadvertent semantic separation, excessive batch dependency), while PaCo/GPaCo specifically tackle class imbalance through parametric rebalancing. Therefore, VarCon and PaCo/GPaCo are orthogonal approaches rather than competing methods. Nevertheless, we provide comparative results by combining VarCon with PaCo/GPaCo's class rebalancing strategy (VarCon+PaCo/GPaCo), leveraging our variational framework's improved representation quality alongside PaCo/GPaCo's handling of imbalanced data:

Table 4: Top-1 accuracy on full ImageNet with ResNet-50.

Table 5: Top-1 accuracy on full CIFAR-100 with ResNet-50.

Due to the considerable computational overhead of PaCo/GPaCo (particularly PaCo), and given time constraints, we conducted experiments for SupCon, VarCon, and VarCon+PaCo/GPaCo combinations, while using reported results from original papers for standalone PaCo/GPaCo. The results demonstrate that VarCon integrates effectively with PaCo/GPaCo, achieving new state-of-the-art performance: 79.94% on ImageNet and 81.29% on CIFAR-100. We leave comprehensive evaluation on truly long-tailed datasets (CIFAR-100-LT, ImageNet-LT) as future work and would be happy to include these results in the revised manuscript.

Regarding novelty (W3)

We respectfully disagree with the characterization that VarCon builds upon SimVAE. While both use variational inference tools, they address fundamentally different problems with distinct objectives and mechanisms:

Table 2: Key differences between SimVAE and VarCon.

As shown in the table, SimVAE is a self-supervised generative framework that models data generation, while VarCon is a novel supervised contrastive learning method that optimizes embedding distributions. The different research goals, learning paradigms, and mathematical formulations establish them as independent contributions addressing distinct problems in representation learning.

Limitation discussion

We thank the reviewer for this suggestion. VarCon reduces computational complexity from $O(B^2)$ to $O(B \cdot C)$ and uses less GPU memory than SupCon (see response to Reviewer gzPQ Q1). Overconfidence is mitigated by our adaptive temperature mechanism (Section 3.2). Calibration under distribution shift is validated through strong transfer learning (Section 4.3, Table 4) and few-shot results (Section 4.2, Table 2). We will include the computational analysis in the revised manuscript and have already discussed broader limitations in Appendix A.1.