An Effective Manifold-based Optimization Method for Distributionally Robust Classification

W4: Illustrations on manifolds.

In Appendix G, we provide four visual illustrations that demonstrate key aspects of our research. We showcase the method's ability to learn meaningful geometric structures through various perspectives:

1, The schematic illustration of our methodology (Figure 9);

2, analysis of manifold dynamics on toy datasets (Figure 10);

3, comparison of learned tangent spaces across different models (Figure 11);

4, visualization of perturbation trajectories on the MNIST dataset (Figure 12).

W5. Computational Cost.

In Appendix F, we compare computational running times across various methods using the same number of epochs (200 epochs). As anticipated, our method requires more time than WDRO because it incorporates additional data manifold information while maintaining a similar workflow. The primary additional computation comes from extracting tangent information through Singular Value Decomposition (SVD) of the Jacobian matrix. To address this, we implemented several optimization techniques detailed in Appendix E, including the randomized SVD method. This method proves particularly well-suited for our approach because, as demonstrated in Figure 2, our model's Jacobian exhibits only a few dominant singular values. This characteristic enables the randomized SVD method to achieve both high efficiency and accuracy in approximating the decomposition.

The experimental results show that our method is 50% slower than the classical WDRO per epoch. However, our method exhibits faster convergence — a natural consequence of working with a more focused uncertainty set (a subset of WDRO's uncertainty set). This faster convergence allows us to implement early stopping and achieve comparable performance with fewer epochs. Specifically, we monitor the robust performance on the validation set during training and stop training when performance stops improving more than 5 epochs.

Finally we achieve similar performance(our model outperforms others) while using similar training time (less than 1.2x). We include this new results in Appendix F.5 of our paper.

Reference

[1] Fang et al., Generalizing Importance Weighting to A Universal Solver for Distribution Shift Problems. NeurIPS'23

[2] Shu et al., Meta-Weight-Net: Learning an Explicit Mapping For Sample Weighting. NeurIPS'19

[3] Fang et al., Rethinking Importance Weighting for Deep Learning under Distribution Shift. NeurIPS'20

[4] Boumal, An introduction to optimization on smooth manifolds, 2023.

[5] Bui et al., A unified Wasserstein distributional robustness framework for adversarial training. NeurIPS'22

We thank the reviewer for the thoughtful comments, and address your concerns below.

W1, The gap between optimal solutions from original and dual problems.

Due to the strong duality property, the primal MWDRO problem in Eq.(5) and its dual formulation in Eq.(6) have identical optimal values.

However, computing exact geodesic distances on general manifolds is often computationally intractable. To address this challenge, Theorem 1 analyzes the approximation ratio of the surrogate loss when using the estimated geodesic distance in our problem. We observe that the optimization problem in Eq.(7) is geodesically strongly concave, which guarantees geometric convergence [4]. This geometric convergence property implies a practical approximation strategy. Lemma 1 shows that by accumulating the optimization steps, we can approximate the geodesic distance up to a constant factor of the true value. Based on this distance approximation, Theorem 1 provides theoretical bounds on the surrogate loss. Specifically, it shows that our computationally feasible approach maintains provable approximation guarantees while avoiding the intractability of exact geodesic distance calculations. This theoretical framework connects mathematical optimality and practical implementation, providing a principled foundation for our manifold-based robust optimization method.

W2, Selection on $\nu$.

The 'dynamic mechanism' for setting $\nu$ proposed by [3] works as follows:

$\nu_{j+1} = \nu_j - \eta_{\nu} \left( \delta - \frac{1}{n} \sum_{i=1}^{n} d_g(q_i, \hat{q_i}) \right).$

Here the $\nu_j$ represents the value of $\nu$ at iteration $j$ in Algorithm 1; $\eta_{\nu}$ denotes the learning rate for updating $\nu$, governing how quickly $\nu$ adjusts in response to the perturbation discrepancy; $\delta$ is the maximum allowable perturbation radius specified in Eq.(7); and $d_g(q_i, \hat{q_i})$ is the geodesic distance between the adversarial example $\hat{q_i}$ and the original input $q_i$, approximated through accumulated steps as established in Lemma 1. Finally, $\frac{1}{n} \sum_{i=1}^{n} d_g(q_i, \hat{q_i})$ represents the batch-averaged (population) perturbation cost across $n$ samples.

The mechanism operates by comparing the average geodesic perturbation against the threshold $\delta$. When the population perturbation fall below this threshold, $\nu$ decreases to permit larger deviations between adversarial and original samples. Conversely, when perturbations exceed $\delta$, $\nu$ increases to constrain the adversarial examples closer to their original counterparts. We include the "dynamic mechanism" in Appendix F.1.

W3, The variance of the added Gaussian white noise. More discussion on related works.

On the variance of the Gaussian noise. The x-axis represents the variance of the additive Gaussian noise, ranging from 0 to 1. (The input data has been normalized.)

Comparison with Related Work. Thank you for your references to the related works [1,2,3], which employ dynamic importance reweighting techniques to handle the distribution shift problems. However, there are fundamental differences between their methods and ours:

1, Different uncertainty set: From the perspective of Distributionally Robust Optimization (DRO), their "uncertainty set" is primarily focused on distributions over the original input data, potentially incorporating validation datasets. The key difference is that our uncertainty set explicitly accounts for perturbed data, which is crucial for scenarios involving data perturbations, such as adversarial attacks. Since these methods do not involve training on perturbed samples, they may show suboptimal performance when facing adversarial examples.

2, Different application scenarios and evaluation metrics: These methods are designed for natural distribution shifts (e.g., covariate shift, label shift) and evaluate performance on datasets exhibiting such shifts. Our method specifically targets robustness against adversarial perturbations and evaluates performance under various attack scenarios (similar to other WDRO works [5]). While both approaches aim to improve model generalization, they optimize for different types of distribution changes.

3, Empirical Performance: Empirically, we observed that methods like GIW [1] are less effective against adversarial examples, despite their strong performance on natural distribution shifts. Experimentally, their model achieves less than 30% accuracy on PGD-attacked ($\epsilon = 0.03$) examples from the CIFAR-10 dataset (same setting with Table 3). Our model demonstrates improved performance against unseen perturbations.

We have expanded the discussion on these points in our related works; please see lines 898–908 for further details.

We thank the reviewer for the thoughtful comments, and address your concerns below.

W1, Re-organized of the paper; More support for the backgrounds.

Thank you for your kind suggestions. In our just uploaded temporary revised version, we have revised some description and added additional visualizations and experimental supports. We will carefully modify our paper’s structure after collecting all the final comments from the reviewers and ACs after this rebuttal phase.

Figure 11 demonstrates that our model successfully captures meaningful geometric structure within the data, learning tangent vectors that correspond to interpretable transformations of objects. These transformations appear to align with sensible variations in parts of the objects. In contrast, other methods fail to extract such structured geometric information from the data.

W3, The advantages of the proposed method compared with [1].

Thanks for this question. In general, our work and [1] have the following two key differences:

1. On manifold learning approach: The paper [1] employs a k-nearest neighbor graph to capture the geometric structure of the dataset, where the shortest paths approximate geodesic distances to guide the transportation of probability mass. Their Geometric Wasserstein DRO framework constrains the distribution support set to the observed data points. On the other hand, our approach learns the data manifold structure through neural network. This continuous, differentiable representation offers several advantages: it captures smooth geometric variations in the data manifold rather than discrete graph approximations, furthermore, it naturally accommodates out-of-sample points through the learned feature extractor.

2. On uncertainty set construction: Their DRO formulation restricts the uncertainty set to distributions over the input data, essentially performing a re-weighting of existing samples without the capacity to incorporate novel data points. Their uncertainty set effectively alleviate the over-pessimism of WDRO, but not well-suited for scenarios involving data perturbations, such as adversarial attack. In contrast, our work specifically addresses the uncertainty introduced by data perturbations. Experimentally, their model achieves less than 30% accuracy on PGD-attacked ($\epsilon = 0.03$) examples from the CIFAR-10 dataset (same setting with Table 3). Our model demonstrates improved performance against unseen perturbations.

Q1, Discuss the effect of $\lambda$ and how to select.

Let us explain the parameters $\lambda_1$ and $\lambda_2$ separately:

In our paper, $\lambda_1$ serve as the coefficient of the Jacobian term in the manifold-guided game $\mathcal{L}^*(\theta)$. As shown in Figures 2 and 8, our method successfully extracts manifold information for $\lambda_1$ values of both $0.1$ and $0.01$. Empirical results show that the performance is not sensitive to the specific choice of the hyperparameter $\lambda_1$ within this range. In our experiments, we set the regularization coefficient of the Jacobian to be $0.01$ unless otherwise noted.

The parameter $\lambda_2$ controls the magnitude of the manifold-guided game $\mathcal{L}^*(\theta)$ when conduct distributional robust optimization. Inspired by the advances linking contrastive learning and the classification tasks [2,3], we incorporate $\mathcal{L}^*(\theta)$ as a regularization term in the MWDRO framework (6). Similar to well-known regularization techniques like L1/L2-norm regularization and dropout, this term, while limiting the neural network's capacity to some degree, ultimately enhances both the robustness and accuracy of the classification.

W2 & Q2, Verify the effectiveness of your proposed method on different model architectures

We have tried some toy neural networks (for synthetic toy datasets, in revised version), LeNet (for MNIST dataset), Resnet-18/32/50 (across different tasks as shown in our experiments). The implementation of our methods does not depend on specific model architecture. It is interesting to observe how our methods perform under more advanced architectures. Given the time limit of rebuttal, we plan to conduct more experimental studies on ConvNext and SWIN as our future work.

Reference

[1] J. Liu et al., Distributionally robust optimization with data geometry[J]. NeurIPS'23

[2] W. Huang et al., Towards the generalization of contrastive self-supervised learning. ICLR’23

[3] J. HaoChen et al., Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss. NeurIPS'21

Q1, In Figure 3, the accuracy of each model is already distinguishable even when the noise size is 0.

This is because robust models often sacrifice performance on clean datasets for achieving robustness [3]. This observation aligns with experiments from previous studies [2,3]. Classical WDRO shows the largest drop in performance on clean datasets as shown in Figure 3 and Table 2,3,4. In fact, this aligns with the “overly pessimistic” behavior we described in the introduction: the model becomes overly focused on certain samples within the uncertainty set, which can end up harming its overall performance. Our method better preserves clean accuracy through manifold constraints as we discussed in “Reply to W1”.

Q2, Is the use of contrastive learning essential in the design of the manifold-guided game?

Contrastive learning (CL) is essential for our method for several theoretical and practical reasons:

1, Theoretically, CL shows strong connection to downstream classification tasks [4,5];

2, it naturally aligns with manifold learning objectives as studied in [6,7] and is complementary to Jacobian regularization.

3, the augmentation operation of CL enhanced manifold density, making it effective across diverse datasets;

4, Additionally, its efficient positive/negative pairing mechanism forces the model to become sensitive to semantic changes in the data, while reduces the number of significant singular vectors. This property improves scalability for large datasets.

A potential alternative for CL might be the auto-encoder. However, the auto-encoder proved less effective in our experiments. Auto-encoder did not perform as well as CL for the downstream classification tasks; Moreover, CL's ability to align positive pairs substantially reduces the number of top singular vectors, as demonstrated in Figure 2, which significantly benefits subsequent random SVD computations (detailed in Appendix E).

Q3, Which phase of training contributes to the increased time required for MWDRO? Further optimize the design of the algorithm to reduce the time like low-rank approximation?

The classicial WDRO method is time-consuming, and incorporating the manifold-guided game does not substantially increase the overall processing time. The main additional computation compared with WDRO comes from extracting tangent information through SVD of the Jacobian. For further details, please refer to “Reply to W3.”

Q4, Further optimize the design of the algorithm to reduce the time like low-rank approximation? For example, by utilizing simplified geometric computations or low-rank approximations.

Thank you for your suggestions regarding acceleration. We have explored several techniques to reduce the time complexity, including the randomized SVD [1]. This approach leverages low-rank approximation, which is well-suited to our model due to the distinctive "spiked" distribution of its Jacobian matrix's singular values, as illustrated in Figures 2 and 8. This rapid decay in singular values enables randomized SVD to achieve both higher accuracy and improved computational efficiency, as discussed in [1]. We provide detailed discuss in Appendix E and present the experimental results in Appendix F.5.

Reference

[1] Halko et al., Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM review, 2011.

[2] Bui et al., A unified Wasserstein distributional robustness framework for adversarial training. NeurIPS'22

[3] Wang et al., Balance, Imbalance, and Rebalance: Understanding Robust Overfitting from a Minimax Game Perspective. NeurIPS'23

[4] Huang et al., Towards the Generalization of Contrastive Self-Supervised Learning. ICLR'23

[5] Haochen et al., Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss. NeurIPS'21

[6] Hu et al., Your Contrastive Learning Is Secretly Doing Stochastic Neighbor Embedding. ICLR'23

[7] Tan et al., Contrastive Learning Is Spectral Clustering On Similarity Graph. ICLR'24

We thank the reviewer for the thoughtful comments, and address your concerns below.

W1, Concerns on the motivation and the threshold selection.

The threshold selection in DRO presents a fundamental trade-off between robustness and accuracy. While our method doesn't completely resolve this dilemma, it significantly alleviates it through manifold constraints, as demonstrated in the enclosed table. Key benefits of our approach for alleviate the over-pessimism feature are summarized as follows:

1, We have larger valid threshold range: the performance of the classical WDRO deteriorates rapidly with even slightly larger thresholds ($\delta$ from 0.01 to 0.02); Our MWDRO maintains performance with larger thresholds due to manifold constraints.

2, We have better Robustness-Accuracy trade-off: Classical WDRO includes irrelevant out-of-manifold samples while our method focuses on semantically meaningful variations. Experimental results in Appendix F show that our MWDRO significantly improve the WDRO (or uni-DRO [2]) method across several datasets and tasks under the same setting. We add more results of MWDRO with different thresholds in Table 2,3,4 of Appendix F.3.

The table below shows that the clean accuracy of the Uni-DRO method drops more significantly than ours when we adopt a larger threshold $\delta = 0.02$.

W2, Add figures that illustrate this concepts.

Thank you for your suggestions. We have added four types of visual illustrations to better demonstrate our method. We showcase the method's ability to learn meaningful geometric structures through various perspectives:

1, The schematic illustration of our methodology (Figure 9);

2, analysis of manifold dynamics on toy datasets (Figure 10);

3, comparison of learned tangent spaces across different models (Figure 11);

4, visualization of perturbation trajectories on the MNIST dataset (Figure 12).

W3, The concern on time complexity.

In Appendix F.5, we compare computational running times across various methods using the same number of epochs (200 epochs). As anticipated, our method requires more time than WDRO because it incorporates additional data manifold information while maintaining a similar workflow. The primary additional computation comes from extracting tangent information through SVD of the Jacobian matrix. To address this, we implemented several optimization techniques detailed in Appendix E, including the randomized SVD method. This method proves particularly well-suited for our approach because, as demonstrated in Figure 2, our model's Jacobian exhibits only a few dominant singular values. This characteristic enables the randomized SVD method to achieve both high efficiency and accuracy in approximating the decomposition [1].

The experimental results show that our method is 50% slower than the classical WDRO per epoch. However, our method exhibits faster convergence — It is rational because MWDRO works with a more focused uncertainty set (a subset of WDRO's uncertainty set). This faster convergence allows us to implement early stopping and achieve comparable performance with fewer epochs. Specifically, we monitor the robust performance on the validation set during training and stop training when performance stops improving more than 5 epochs.

Finally we achieve similar performance(our model outperforms others) while using similar training time. We include this new results in Appendix. F.5 ''Time comparison'' of our paper.

We deeply appreciate your time and effort in reviewing our paper and providing insightful and constructive comments. We are glad to hear that our responses address your concerns. If you have any further suggestions or questions, we will make every effort to address them.

W4, How to do Exp operator to retract to the data manifolds?

In practice, when the exponential map $Exp_x(v)$ lacks a closed-form expression, it is commonly approximated with a retraction map $R_x(v)$. As suggested in [3], the linear retraction $R_x(v) = x + v$ is widely used, as it is a first-order approximation of $Exp_x(v)$. In this work, we adopt this linear approximation, as noted in line 462-463.

W5, The concern on time complexity

In Appendix F.5, we compare computational running times across various methods using the same number of epochs (200 epochs). As anticipated, our method requires more time than WDRO because it incorporates additional data manifold information while maintaining a similar workflow. The primary additional computation comes from extracting tangent information through SVD of the Jacobian matrix. To address this, we implemented several optimization techniques detailed in Appendix E, including the randomized SVD method. This method proves particularly well-suited for our approach because, as demonstrated in Figure 2, our model's Jacobian exhibits only a few dominant singular values. This characteristic enables the randomized SVD method to achieve both high efficiency and accuracy in approximating the decomposition [4].

The experimental results show that our method is 50% slower than the classical WDRO per epoch. However, our method exhibits faster convergence — a natural consequence of working with a more focused uncertainty set (a subset of WDRO's uncertainty set). This faster convergence allows us to implement early stopping and achieve comparable performance with fewer epochs. Specifically, we monitor the robust performance on the validation set during training and stop training when performance stops improving more than 5 epochs.

Finally we achieve similar performance(our model outperforms others) while using similar training time (less than 1.2x). We include this new results in Appendix F.5 ''Time comparison''.

W6, The benefit of performing DR on data manifolds; The visualization of trajectory.

We have expanded our analysis by incorporating additional ablation studies and visual evidence.

The experiments in Figure 7 demonstrate the importance of performing the distributional optimization on data manifold. In Phase 1 of Algorithm 1, we initially train the model for $N_0$ epochs using $L^*(\theta)$ (Eq.(8), which combines the CL and Jacobian regularization) to learn the data manifold structure before implementing the MWDRO algorithm. Our ablation study in Figure 6 varies $N_0$ and reveals three key findings:

1, both robust and clean accuracy remain low for small values of $N_0$, indicating insufficient manifold learning results in suboptimal performance;

2, Besides, the performance improves as $N_0$ increases as the model extracts more refined geometric information;

3, the improvement plateaus once the model adequately captures the manifold structure ($N_0 > 40$). These observations demonstrate that DRO tends to be overly pessimistic without proper manifold constraints, highlighting the crucial role of accurate manifold information for effective robust optimization.

For the visualization aspect, Appendix G presents four visual illustrations that demonstrate key aspects of our research. We showcase the method's ability to learn meaningful geometric structures through various perspectives:

1, The schematic illustration of our methodology (Figure 9);

2, analysis of manifold dynamics on toy datasets (Figure 10);

3, comparison of learned tangent spaces across different models (Figure 11);

4, visualization of perturbation trajectories on the MNIST dataset (Figure 12).

Reference

[1] Wang and Isola, Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere. ICML'20

[2] Boumal, An introduction to optimization on smooth manifolds, 2023.

[3] Bécigneul and Ganea, Riemannian adaptive optimization methods, ICLR'19.

[4] Halko et al., Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM review, 2011.

We thank the reviewer for the thoughtful comments, and address your concerns below.

W1, Why can we use the linear span of top singular vectors of the Jacobian matrix of $g$ to characterize the tangent space of data manifolds?

The Jacobian matrix $J_\mathbf{g}(x)$, which contains the partial derivatives of the features with respect to the input data, uncovers how the representation changes in response to input variations. Through training with the manifold-guided game, the model maintains strong responses along directions of semantic variation in the tangent while exhibiting weaker responses in other directions. This results in a distinctive “spiked” distribution of the Jacobian matrix’s singular values (as shown in Figure 2 and 8), where the top singular vectors naturally correspond to the semantic variate directions in the tangent.

The role of the game mechanism: The contrastive learning component encourages the model to produce discriminative representations for samples with distinct semantics, while Jacobian regularization suppresses feature variations caused by perturbations of $x$ in all directions. Together, these objectives enable the model to effectively capture the intrinsic structure of the data. In the revised version, we include additional visualizations to illustrate this effect more intuitively in Appendix G.

W2, Why the game in Section 4.2 helps extract tangent information?

The effectiveness of our game mechanism can be understood through the lens of representation learning theory. Following [1], the InfoNCE loss of the contrastive learning can be decomposed into two key effects: 1, Uniformity, the representations of the negative pairs are pushed further apart. This encourages maximal change in $\mathbf{g}(\cdot)$ along semantically meaningful directions within the manifold's tangent space. 2, Alignment, This brings representations of semantic-preserving augmentations closer together. While the Alignment effect shares some purpose with Jacobian regularization, the Uniformity effect is crucial for identifying the principal directions of variation on the manifold. Together, they shape the encoder $\mathbf{g}(\cdot)$ to have two essential properties, which are summarized as follows:

1, Sensitivity. The encoder $\mathbf{g}(\boldsymbol{x})$ should change most when the sample $\boldsymbol{x}$ moves along the semantic variant directions within the data manifold;

2, Insensitivity. The encoder $\mathbf{g}(\boldsymbol{x})$ should change minimally when moves off the manifold (i.e., $\mathcal{M}^\perp$), or towards the Semantic-Invariant region (i.e., $\mathcal{M}_{\text{SI}}(\mathbf{x})$). These properties ensure that the top singular vectors of $J_g(x)$ provide an effective approximation of the semantic variate directions within the tangent as discussed in our response to W1.

Further interpretations can be found in Appendix B, with visual illustrations provided in Appendix G. We believe our approach is novel and hope our work can inspire the further exploration to other optimization problems on manifolds where closed-form expressions are unavailable.

W3, Why can we use the $pt^t$ in Alg 2 to approximate the geodesic distance from $q_t$ to $q_0$？

Computing exact geodesic distances on general manifolds is computationally intensive and often intractable. It is intuitive to approximate the trajectory length using the accumulated discrete step sizes in practice. Our key insight is to leverage the geodesically strong concavity property of the surrogate loss in Eq.(7) to provide theoretical guarantees for this approximation. This strong concavity ensures geometric convergence in optimization, which means the step sizes decrease at a rate following a geometric sequence [2]. This guarantees that the estimated geodesic distance approximates the true geodesic distance within a constant factor, as stated in Lemma 1.

We thank the reviewer for the thoughtful comments, and address your concerns below.

A1. Thanks for this question. To explain why we don't explicitly construct a local defining function $h(x) = 0$ for extracting the tangent information, we refer to the contractive auto-encoder methodology [2] as an example. In [2], the authors constructed a topological atlas of charts characterized by the principal singular vectors of the representation mapping's Jacobian. Even though the neural network used in [2] is relatively simple compared with the more complicated networks used nowadays (note [2] is published in 2011), it already demonstrated good results. This line of approach bypasses the challenging task of obtaining local defining functions for manifolds by directly extracting tangent information, a methodology proven effective and efficient in both previous works and ours.

Also, we would like to mention that the concept of using neural networks to extract tangent space information has been validated in several previous articles. For example, Figure 6 in [1] and Figure 1 in [2] clearly demonstrate the extracted tangent vectors. Our experimental results in Appendix G (Visualizations) provide additional evidence supporting this. Furthermore, we validated the effectiveness of this approach in addressing the MWDRO problem of concern.

A2. Please note that precisely acquiring the data manifold remains an open challenge, and previous studies suggest that neural network-based approaches could provide promising approximations. It is well known that the columns of the Jacobian matrix $J(x)$ represent partial derivatives along each input dimension. In high-dimensional spaces, using these partial derivatives directly would yield an excessive number of potential tangent vectors, with many directions being redundant or capturing noise. For instance, in [2], the authors also used the span of the top singular vectors to approximate the data manifold's tangent information.

A3. Previous studies [1,2] have shown that it is feasible to leverage the neural network's powerful representational capability to approximately extract the tangent information of this manifold. Building on these insights, we employed contrastive learning techniques to extract the semantic variation subspace within the tangent space, requiring fewer top singular vectors (in Figures 2 and 8). We validated the effectiveness of this approach in addressing the MWDRO problem of concern.

For future work, we hope our currently proposed method could inspire further explorations of other optimization problems on manifolds, where closed-form expressions are unavailable.

A4. As you correctly point out, we can project the gradient of the geodesic distance onto the tangent space. Please note that the gradient of the squared geodesic distance $\nabla_{q^t} d^2_g(q^t, q)$ can be expressed as $-2 d_g(q^t, q) \hat{u}$, where $\hat{u}$ is the unit-length initial tangent vector of the geodesic from point $q^t$ to point $q$. In our experiments, the $\hat{u}$ vector is approximated by $\mathrm{Proj}^{\tau_0}_{M} \Big( \frac{q - q^t}{\| q^t - q \|_2} \Big)$.

A5. Thanks for this question, and let us explain it in more detail. Following the work [3], when $exp_x(v)$ lacks a closed-form expression, it is common practice to employ the map $R_x(v) = x+v$ as a computationally efficient approximation. Since a manifold is locally homeomorphic to Euclidean space in a neighborhood of each point, we can approximate the exponential map $Exp_{q_t}(\alpha \nabla^M f(q))$ using $q^{t+1} = q^t + \alpha\nabla^M f(q)$ with a small step size $\alpha$ (to approximate the gradient flow on the manifold).

We provide several visualizations to demonstrate these effects in our experiments. Moreover, some other empirical validation in previous works can be found [1,2]. For example, in Figure 6 of [1], the points generated by adding tangent vectors to manifold points successfully capture meaningful transformations (such as rotation and scaling) of the original images. These empirical results suggest that such an approximation maintains the data's local geometric structure, making it suitable for practical applications.

Reference:

[1] Patrice Y Simard, Yann A LeCun, John S Denker, and Bernard Victorri. Transformation Invariance in Pattern Recognition – Tangent Distance and Tangent Propagation. NN: Tricks of the Trade, 2nd edn., LNCS 7700, 2002)

[2] Salah Rifai, Yann N Dauphin, Pascal Vincent, Yoshua Bengio, and Xavier Muller. The manifold tangent classifier. Advances in neural information processing systems, 2011.

[3] Bécigneul and Ganea, Riemannian adaptive optimization methods, ICLR'19.

We thank the reviewer for the thoughtful comments, and address your concerns below.

W1, Whether the effectiveness of the algorithm depends on specific data?

Our method is well-suited for data with low-dimensional manifold structures, such as natural images. This is because the intrinsic characteristics of such data (e.g., semantic variations) are often concentrated in a few key directions, which can be effectively captured by our manifold-guided game. A potential concern is whether our manifold-guided game remains effective for highly complex or sparse data distributions. We think the augmentation operation of CL might mitigate this concern. On the on hand, the augmentation produces a potentially infinite set of samples on the data manifold. On the other hand, the relationships among the augmented data help uncover the intrinsic geometric structure of the data, please refer to [1,2] for the justification.

In our work, we demonstrate the effectiveness of our approach through extensive experiments across various datasets and tasks. MWDRO achieves strong performance across a range of datasets, including CIFAR-10/100, Tiny-ImageNet, and medical images. This robust performance across different datasets and tasks suggests that our approach effectively capture the underlying manifold structures in many real-world datasets.

Reference

[1] Wang et al., Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlap. ICLR'22

[2] Haochen et al., Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss. NeurIPS'21