PCA++: How Uniformity Induces Robustness to Background Noise in Contrastive Learning

The assumptions 1-3 are somewhat restrictive, and may be unrealistic in real scenarios, in particular the orthogonality in Assumption 1. The authors should discuss the dependency and sensitivity of the results on these assumptions, either by theoretical analysis or numerical experiments.

A: We thank the reviewer for their insightful questions regarding the assumptions of our model. We agree that discussing their scope and necessity is crucial. We address the main points below and will add these clarifications to the revised manuscript.

• On the orthogonality assumption: (Assumption 2.1) We acknowledge that this is a strong assumption, made primarily for analytical tractability. Our framework's core mechanism is, however, robust to its violation. The key insight is that even if signal and background subspaces overlap, the contrastive energy (as analyzed in Lemma C.1) of the shared directions remains strictly positive, while that of pure background directions is zero. This allows PCA++ to distinguish the full signal space from the background. The main effect of overlap is a reduction in the generalized eigenvalue for the shared directions, but they remain detectable. For a more detailed technical sketch and experimental validation, we respectfully refer the reviewer to our response to Reviewer [9ppt].

• On the Gaussian latent factor assumption: (Assumption 2.2) This assumption was also made for analytical convenience, as it allows for the clean derivation of exact constants and closed-form error rates. However, we expect the core results to hold more generally for sub-Gaussian distributions. Many of the key results from random matrix theory that we rely on (e.g., in Lemmas D.4 and D.5) have well-known extensions beyond the Gaussian case, often requiring only a few finite moments. We chose to present the Gaussian case for clarity and precision, but relaxing this assumption is a natural direction for future theoretical work, which would likely involve more complex proofs.

  To provide strong empirical evidence for this claim, we have run new simulations where the Gaussian latent factors $(w\_i, h\_i)$ and noise $(\epsilon\_i)$ from Assumptions 2.2 and 2.3 were replaced with samples from a standardized $Beta(2,2)$ distribution (a symmetric, bounded, non-Gaussian distribution). We repeated the experiments from Sec E.4 (Figure 4) under this new setting.

  The results below show that the empirical performance of PCA++ under Beta-distributed noise continues to align almost perfectly with our theoretical predictions, which were derived under the Gaussian assumption.

  Fixed Aspect Ratio with Beta distribution

  Aspect Ratio	0.1	0.3	0.5	0.7	0.9	1.1	1.3	1.5	1.7	1.8
  PCA++	0.095	0.195	0.244	0.266	0.295	0.361	0.385	0.381	0.402	0.412
  PCA++ theory predicted	0.104	0.179	0.229	0.268	0.301	0.330	0.356	0.379	0.400	0.410

  Growing-spike regime with Beta distribution

  Aspect Ratio	0.1	0.3	0.5	0.7	0.9	1.1	1.3	1.5	1.7	1.8
  PCA++	0.112	0.171	0.223	0.260	0.296	0.331	0.348	0.360	0.389	0.409
  PCA++ theory predicted	0.100	0.171	0.218	0.256	0.287	0.315	0.339	0.361	0.381	0.391

  This remarkable consistency provides strong evidence that the Gaussian assumption is a technical choice for analytical clarity rather than a strict requirement for the validity of our results, which appear to exhibit universality. We will add these new findings to the appendix to empirically support the robustness of our framework.

• On other assumptions: (e.g., Assumption 4.1, 4.3) We have also provided a detailed justification for the high-dimensional assumptions regarding the BBP threshold and distinct eigenvalues in our response to Reviewer [r13i]. The key takeaway is that the BBP threshold is a fundamental limit of PCA-based detection, while the distinct eigenvalue assumption is a technical convenience that we validate empirically.

We believe these clarifications properly scope our theoretical contributions, and we will incorporate this discussion into the final manuscript. Thank you again for your valuable feedback.

The authors should discuss the computational cost of the proposed framework, and it scalability to large datasets.

A: Thanks for the suggestion. We will add a detailed analysis of the computational cost and scalability of PCA++ to the revised manuscript.

Computational Complexity

The computational cost of PCA++ has two main components:

1. Covariance matrix formation: We compute the sample covariance $S\_n$ and the contrastive covariance $S\_{n}^{+}$. For data matrices of size $n \times d$ (samples × features), forming these $d \times d$ matrices requires matrix multiplications with a complexity of $O(nd^{2})$. In the common high-dimensional setting where $d\gg n$, this can be optimized to $O(dn^{2})$ by first computing the $n\times n$ Gram matrix.

2. Generalized eigenvalue problem (GEP) solution: The cost is dominated by the initial truncated eigendecomposition of $S\_n$ to rank $s$. Using an iterative solver like the Implicitly Restarted Lanczos Method (IRLM), as implemented in scipy, this step has a complexity of approximately $O(sd^{2})$. Since the truncation rank $s$ is typically much smaller than $d$, this step is highly efficient.

Overall, the complexity is comparable to standard PCA, primarily driven by the feature dimension $d$.

Empirical Scalability

To demonstrate its practical performance, we benchmarked the runtime of PCA++ while varying the number of samples ($n$) and features ($d$). The results confirm our theoretical analysis.

Computational Cost of PCA++ (in seconds)

Setup: Truncation rank $s=10$, top $k=5$ eigenvectors estimated. Benchmarked on an Intel Xeon CPU @ 2.20GHz.

The benchmarks show that the runtime is dominated by the feature dimension $d$, scaling quadratically, while it scales nearly linearly with the number of samples $n$. This profile makes PCA++ computationally feasible for typical high-dimensional datasets with tens of thousands of features, confirming its practical scalability.

We thank Reviewer [9ppt] for their constructive feedback and for recognizing the paper's strengths. We agree with their assessment of the weaknesses and will address each point to significantly improve the manuscript.

On the Orthogonality Assumption:

The assumption which requires orthogonal signal and background subspaces is quite strong and may not hold in many real-world applications.

A: We thank the reviewer for raising this crucial point. We agree this assumption is strong and adopted it for analytical tractability. However, the core mechanism of PCA++ is robust to violations of this assumption.

Why the core mechanism is robust to overlapping subspaces: The key insight, enabled by the linearity of our factor model (Eq. 2.1), is that we can analyze the behavior of the covariance matrices in a shared basis. Even with overlap, the contrastive covariance $S\_{n}^{+}$ isolates signal components, while the standard covariance $S\_n$ accumulates variance from both signal and background in the shared directions.

Let's sketch this for a simple case. Because the model is linear, we can define an orthonormal basis that accounts for the overlap. Suppose span(A) and span(B) share a single direction $v\_0$. We can decompose the subspaces as:

• Signal space: span(A) is spanned by $\\{ v\_0, v\_{A,1}, ..., v\_{A,k-1} \\}$, where $v\_0$ is the shared part and $v\_{A,i}$ are the pure signal directions, orthogonal to $v\_0$.

• Background space: span(B) is spanned by $\\{ v\_0, v\_{B,1}, ..., v\_{B,m-1} \\}$, where $v\_{B,j}$ are the pure background directions.

The population covariances then become:

• Contrastive covariance: $E[S_{n}^+] = \lambda\_{A} v\_{0} v\_{0}^\top + \sum\_{i=1}^{k-1} \lambda\_{A, i} v\_{A, i} v\_{A, i}^\top.$ Here, $\lambda\_{A,0}$ is the signal variance in direction $v\_{0}$. Thus, contributions from the pure background directions $\\{v\_{B,j}\\}$ are still averaged out. The resulting expectation remains spanned only by the signal-related directions $\\{ v\_0, v\_{A,1}, ..., v\_{A,k-1} \\}$. The shared direction $v\_{0}$ is not cancelled.

• Standard covariance: $E[S_{n}] = (\lambda\_{A, 0}+\lambda\_{B, 0}) v\_{1} v\_{1}^\top + \sum\_{i=1}^{k-1} \lambda\_{A, i} v\_{A, i} v\_{A, i}^\top + \sum\_{i=1}^{m-1} \lambda\_{B, i} v\_{B, i} v\_{B, i}^\top.$ Here, $\lambda\_{B, 0}$ is the background variance in direction $v\_{0}$. The variance in the shared direction $v\_{0}$ is amplified by both signal and background components.

Intuition -- why PCA++ is robust to violations of orthogonality assumption: The robustness of PCA++ stems from how the generalized eigenvalue problem $S\_{n}^{+} v = \lambda S\_{n} v$ interacts with these modified covariance structures. We can analyze this through the lens of the asymptotic contrastive energy for each direction, as defined in our analysis for Lemma C.1.

• Pure background directions ($v\_{B,j}$): The contrastive energy $v\_{B,j}^T S\_{n}^{+} v\_{B,j}$ remains asymptotically zero, so these directions are filtered out.
• Shared direction ($v\_0$): The contrastive energy $v\_0^T S\_{n}^{+} v\_0$ is strictly positive due to the signal component. However, its variance in the standard covariance is now amplified by both signal ($\lambda\_{A,0}$) and background ($\lambda\_{B,0}$) components.

The resulting contrastive energy for shared direction will be smaller than that of a pure signal spike, but it will still be bounded away from zero. Therefore, the PCA++ objective still detects the shared direction as part of the signal space. The fundamental mechanism—isolating all directions with non-zero contrastive energy—remains intact.

Empirical Validation with Overlapping Subspaces:

To provide strong empirical evidence, we ran new simulations with non-orthogonal signal and background subspaces and will add these to the Appendix. We ran new simulations based on the setting in Sec E.4, but introduced overlap between the signal and background subspaces. We aligned two background directions with the two weakest signal directions, creating a two-dimensional shared subspace. We tested this under two background noise levels.

• Signal variances: [50, 25, 20, 15, 10]
• Moderate noise background: [500, 400, 300] (pure background) + [25, 12.5] (shared background)
• Large noise background: [500, 400, 300] (pure background) + [100, 50] (shared background)

The results for the fixed aspect ratio regime are shown below. For this demonstration, we compare against our original theoretical predictions (derived under orthogonality).

Fixed Aspect Ratio with Moderate Overlapping Noise:

Fixed Aspect Ratio with Large Overlapping Noise

Growing-spike regime with Moderate Overlapping Noise

Growing-spike regime with Large Overlapping Noise

Conclusion: The results show that PCA++ is remarkably robust, even when the orthogonality assumption is violated. In the moderate overlapping noise case, the empirical error of PCA++ continues to track the theoretical predictions remarkably well. This demonstrates that when the background variance in the shared subspace is not excessively large, the impact of the overlap is minimal.

In the large overlapping noise case, we observe a slight increase in estimation error, as expected. This is because the background noise in the shared directions becomes strong enough to reduce the effective signal-to-noise ratio, making recovery more challenging. Nevertheless, even in this challenging scenario, PCA++ remains stable and successfully recovers the signal subspace with controlled error, confirming that perfect orthogonality is not a practical prerequisite for our method's success. We will add this discussion and the full experiment to the paper.

On Selecting Hyperparameter $s$:

The strategy for selecting the hyperparameter s is not reasonable.

A: We thank the reviewer for this important practical point. We agree that our paper must provide clearer guidance on choosing the truncation rank $s$.

The choice of $s$ in truncated PCA++ controls a fundamental trade-off between the stability of the generalized eigenproblem and the effectiveness of the uniformity constraint. We will add a new subsection to the Appendix with practical guidelines for selecting s, centered on the following trade-off:

• A small $s$ ensures that the inverse of the truncated covariance $(S\_n)\_s$ is well-conditioned, promoting numerical stability. However, it may discard dimensions needed to enforce uniformity effectively, potentially biasing the result.
• A large s enforces uniformity over a larger subspace but risks instability if $S\_n$ is ill-conditioned, as including directions with near-zero eigenvalues can amplify noise. As shown in our experiments (Figure 2, left).

Based on this trade-off, we propose to combine the following practical strategies for selecting $s$:

• Information-based criterion (lower bound): As in standard PCA, one can determine a minimum $s$ by examining the cumulative variance explained by the eigenvalues of the sample covariance $S\_n$. For example, choose $s$ large enough to capture a significant portion (e.g., 90%) of the total variance. This ensures the uniformity constraint operates on the most meaningful directions.

• Stability-based criterion (upper bound): To ensure numerical stability, one can monitor the condition number of the truncated matrix $(S\_n)\_s$, which is the ratio of its largest to its $s$-th eigenvalue $(\lambda\_1(S\_n) / \lambda\_s(S\_n))$. One should choose $s$ such that this ratio remains below a reasonable threshold, avoiding severe ill-conditioning.

In our experiments (e.g., Figure 2, right), we found that PCA++ is robust across a reasonable range of $s$. We will incorporate these guidelines into Section 4 of the paper to make the method more accessible.

On Limitations and Typos:

The limitations of PCA++ are not clearly discussed in the paper.

A: We agree with the reviewer that this was an omission. As detailed in our response to Reviewer [uGvG], we will add a dedicated Limitations section to the paper, focusing on the assumptions underpinning our theoretical model.

Some typos.

A: We will carefully proofread the manuscript and correct all identified errors.

We are confident these substantial revisions will fully address the reviewer's concerns and strengthen the paper. We are grateful for their thoughtful feedback.

Dear Reviewer,

As the discussion period draws to a close, we would be sincerely grateful if you could kindly review our clarifications. If there are any remaining questions or concerns, we would be happy to address them.

If our responses have adequately addressed your queries, we would greatly appreciate it if you could consider revisiting your evaluation in light of the clarifications provided.

Thank you once again for your time and thoughtful review.

We sincerely thank Reviewer [uGvG] for their detailed feedback and for identifying a problem in our paper's positioning. We agree that our failure to cite and discuss the previous "contrastive PCA" literature (e.g., Abid et al., 2017; Li et al., 2020, Boileau et al. 2020, Zhou et al. 2023, Zhang et al. 2025, etc) was a major oversight. We see now how this created an inaccurate impression of our work's novelty, and we are grateful for the chance to correct this problem and clarify our specific contributions.

Our response is structured to directly address the reviewer's two primary concerns:

1. Fundamental difference in problem formulation: Our work (PCA++) addresses a very different, and arguably more common, problem setting than the previous contrastive PCA literature (cPCA) cited by the reviewer.

2. Novelty and significance of our contribution: Our core contribution is a novel and rigorous high-dimensional analysis that yields new theoretical insights into the role of uniformity, not the invention of the uniformity concept itself.

1. Clarifying our problem setting vs. cPCA

The reviewer's main concern stems from a misunderstanding of the problem setting, which was caused by our poor positioning. We will rectify this in the manuscript. There is a fundamental distinction:

Input Data:

• cPCA: Two distinct datasets -- a foreground/target dataset ($X$) containing both signal and background, and an explicit background-only dataset ($Y$).

• PCA++: Paired "positive views" $(X, X^{+})$ that share a signal but have different background instances.

Objective:

• cPCA: Find directions of variance in $X$ that are not present in $Y$. (e.g., via covariance subtraction $\Sigma\_{X} - \alpha \Sigma\_{Y}$).

• PCA++: Find directions that are shared/invariant between the positive views $(X, X^{+})$

Typical Scenario:

• cPCA: Comparing cases vs. controls; comparing two different experimental conditions.

• PCA++: Self-supervised learning from data augmentations (standard setting of modern contrastive learning, e.g., SimCLR, MoCo); analyzing paired measurements (e.g., our single-cell data).

This distinction is critical. The methods cited by the reviewer are not applicable to our problem setting, as they require access to a pure background dataset, which is unavailable in the standard positive-pair contrastive learning framework. Our work is expressly designed for this latter, widely used scenario.

Action: We will revise our Introduction and Related Work sections. We will begin by clearly delineating these two distinct "contrastive" paradigms. We will properly cite and discuss the foreground-background cPCA literature (Abid et al., etc.) and explicitly state that our work addresses the positive-pair setting, thereby correctly positioning our contribution within the context of modern contrastive learning [8, 10, 41].

2. On the novelty and significance of our contributions

The reviewer suggests our contribution is questionable because the "uniformity constraint" is an existing idea. We respectfully but firmly disagree. Our novelty lies not in proposing the concept of uniformity, but in the following non-trivial contributions:

A. Principled Motivation and Failure Analysis: We do not simply add a uniformity constraint. Our paper's narrative is a core part of our contribution:

• We first establish a natural baseline, alignment-only pca+.

• We then provide a novel theoretical failure analysis (Thm 3.4), proving that this intuitive approach provably fails when background noise is strong -- a key insight that motivates the need for a stronger regularizer.

• This directly leads to PCA++, where the uniformity constraint is introduced as a principled and necessary remedy to this specific, proven failure mode.

B. New, Rigorous High-Dimensional Theory: Our primary contribution is a deep theoretical analysis that goes far beyond prior work.

• Exact Asymptotic Limits: Instead of loose non-asymptotic bounds, we are the first to provide exact, closed-form asymptotic limits for the subspace recovery error using powerful tools from random matrix theory (Thm 4.2, 4.3). This offers a much sharper and more precise understanding.

• Proving Why Uniformity Works: Our theory provides the first quantitative proof of how uniformity confers robustness. By comparing the error formula for PCA+ (which depends on background strength) with that of PCA++ (which is provably independent of background strength), we demonstrate a precise, mathematical mechanism. While Wang and Isola [41] noted that InfoNCE implicitly encourages uniformity, our work provides a rigorous explanation for why this is so beneficial in the presence of structured noise.

Action: We will sharpen the language in our abstract and introduction to emphasize that our core technical novelty lies in the exact high-dimensional characterization of a uniformity-constrained estimator and the new insights this theory provides into the noise-suppressing role of uniformity in contrastive learning.

3. On Discussing Limitations

We agree that a clear discussion of limitations is essential and will add a dedicated "Limitations and Future Work" section to the revised manuscript.

The primary limitation of our work—which is also its key analytical strength—is the adoption of a linear contrastive factor model. While real-world data often exhibits non-linearities, this deliberate choice was crucial for achieving our paper's main goal: to provide a precise, quantitative explanation of how uniformity provides robustness in contrastive learning.

This tractable linear structure enabled us to:

1. Derive exact, closed-form asymptotic limits for the subspace estimation error (Theorems 4.2 and 4.4), a stronger result than the looser upper bounds common in more general settings.
2. Precisely isolate and demonstrate the mechanism by which the uniformity constraint filters out strong, structured background noise—a level of clarity that would be difficult to achieve with more complex, non-linear models.

Our work thus serves as a foundational, theoretical study that provides concrete insights into the role of uniformity.

Future Work: An important next step is to extend this framework to handle non-linearities. As the reviewer might anticipate, a natural direction is to develop a kernelized version of PCA++, which we plan to explore in future work. This would involve applying the same principles of contrastive alignment and hard uniformity in a reproducing kernel Hilbert space, potentially offering the same robustness benefits for non-linear data.

We will add this discussion to the paper to clearly scope our contributions and outline promising future directions.

Action: We will add this discussion to the paper to clearly scope our contributions and outline promising future directions.

We are confident that once our paper is revised to properly position our work, the novelty and significance of our contributions will be clear. Our work tackles a different problem than prior cPCA and provides a new, deep theoretical analysis that has not been previously shown. We believe the reviewer's concerns, while stemming from a real flaw in our initial manuscript's presentation, are fully addressable. We hope they will re-evaluate our work based on these clarifications and planned revisions.

Experimental results (Subspace Error)

The results are provided in the following tables. As the results clearly show, the cPCA adaptation is highly unstable even with its optimal hyperparameter setting $\alpha=1$. This, combined with the conceptual issue in the truncated cPCA++ approach, demonstrates that these methods are not well-suited for this problem. In contrast, PCA++ method remains stable, highlighting the robustness of our GEP formulation. We will add these results to the final manuscript.

Results with with Moderate noise background

Results with Large noise background

2. On the Relation to Canonical Correlation Analysis (CCA):

We thank the reviewer for this question, as the connection to Canonical Correlation Analysis (CCA) is insightful and helps clarify the unique contribution of PCA++.

While both methods aim to find shared structure between paired datasets, they differ crucially in their objectives and constraints, leading to significant performance gaps in noisy, high-dimensional settings.

Objective functions and constraints:

• Standard CCA: Seeks two distinct projection matrices, $U$ and $V$, that maximize the correlation between the projected views $U^T X$ and $V^T X^+$. The objective for the leading component is: $\max\_{u,v} \frac{u^T (X^T X^+) v}{\sqrt{(u^T (X^T X) u) (v^T (X^{+T} X^+) v)}}$ The key instability lies in the denominator, which normalizes by variance from both views ($X$ and $X^{+}$). In our problem setting, both views contain large, independent background and noise components, making this normalization highly susceptible to noise amplification. Although one can also revise CCA for the positive-pair setting by imposing constrain $U=V$, the objective becomes a Generalized Rayleigh Quotient, which, to our knowledge, does not have the same closed-form solution as standard CCA.

• PCA++: Our formulation makes two critical changes that enhance stability:

  • Shared projection space: It enforces $U=V$, searching for a single, shared subspace that captures the signal common to both views.
  • Robust normalization: Instead of normalizing by the noisy variance from both views, PCA++ maximizes the shared covariance $v^T (X^T X^+) v$ subject to a hard uniformity constraint on only one view $v^T (X^T X) v = 1$. This constraint acts as a powerful regularizer, anchoring the solution to a more stable variance structure and preventing distortion from the background and noise present in the second view $X^+$.

Empirical Performance:

This difference in formulation has a dramatic impact on performance. As our new experiments (provided in our response to the previous question) demonstrate:

• Standard CCA is highly unstable in the settings we study. It fails to recover the signal, with its performance collapsing as dimensionality or noise increases.

• PCA++ remains stable and robust, successfully recovering the signal subspace across all tested regimes.

This confirms that our specific choice of a shared projection space and a single-view uniformity constraint is essential for reliable signal recovery in this problem setting. We will add a discussion of this important relationship with CCA to the revised paper.

3. On the "Contrastive Factor Model" and i.i.d. Views:

We thank the reviewer for this observation about our modeling assumptions. The reviewer notes that under Assumptions 2.2 and 2.3, $x\_i$ and $x\_i^+$ are drawn from the same marginal distribution. This is a deliberate choice that aligns with standard theoretical models of contrastive learning.

The goal in contrastive learning is to learn representations that are invariant to nuisance variations while being sensitive to shared semantic content. Our model captures this by linking $x\_i$ and $x\_i^+$ through a shared latent signal $w\_i$. The "contrast" arises because they are conditioned on different instances of background and noise ($h\_i$ vs. $h\_{i}^{+})$. The task is to recover the signal $A$ that is invariant across these instances. Considering the positive pair $(x\_i, x\_i^+)$ as a single sample from a joint distribution with identical marginals is a standard way to formalize this (e.g., as in Wang and Isola, [41]).

The reviewer suggests that assuming $x\_i$ and $x\_{i}^{+}$ come from different distributions might seem more realistic. However, such a setting could paradoxically simplify the learning problem. For example, if the background subspace for $x\_{i}^+$ (say, $B^+$) were orthogonal to the background subspace for $x\_i$ (say, $B$), then the cross-term in the contrastive covariance would be exactly $(Aw\_i+Bh\_i)^T(Aw\_i+B^{+}h\_{i}^+) = AA^T$. In this scenario, $S\_{n}^+$ would be a cleaner estimator of $AA^T$ than in our current, more challenging setting, where both views share the same background space $B$. Therefore, our assumption of identical marginal distributions is not an oversimplification but rather a standard and rigorous formulation for the problem of learning invariances from different augmentations or views of the same underlying data type. We will clarify this motivation in the paper.

We thank Reviewer [uGvG] for their thoughtful re-evaluation and helpful follow-up questions. We address each of the additional points below.

1. On Using cPCA and cPCA++ as a Baseline:

Synthesizing a foreground $x\_{f} = (x + x^{+})/2$ and background $x\_{b} = (x - x^{+})/2$ to use with cPCA-like methods is an interesting approach. While this is theoretically sound at the population level, our new experiments, prompted by the reviewer's comment, reveal that it can be less stable in finite-sample settings compared to our proposed PCA++.

At the population level, the signal is perfectly isolated. Under the contrastive factor model, the population covariances of the synthesized data are:

• $\Sigma\_f = \mathbb{E}[x\_f x\_f\^{T}] = AA\^{T} + \frac{1}{2}BB\^{T} + \frac{1}{2}I\_d$

• $\Sigma\_b = \mathbb{E}[x\_b x\_b^{T}] = \frac{1}{2}BB^{T} + \frac{1}{2}I\_d$

Therefore, the difference $\Sigma\_f - \Sigma\_b = AA\^T$ perfectly recovers the signal covariance. The practical challenge, however, arises from finite-sample estimation error. Define foreground matrix $X\_{f} = (X + X^{+})/2$ and background matrix $X\_{b} = (X - X^{+})/2$. The sample covariances of the synthesized data are $\hat{\Sigma}\_f = X\_{f}^T X\_{f}/n$ and $\hat{\Sigma}\_b = X\_{b}^T X\_{b}/n$, respectively.

• Regarding cPCA (Abid et al., 2017): The reviewer's proposed approach relies on the matrix subtraction $\hat{\Sigma}\_{f} - \alpha \hat{\Sigma}\_{b}$, this subtraction of two large, noisy matrices can amplify noise, potentially overwhelming the true signal and leading to unstable eigenvectors.

• Regarding cPCA++ (Salloum and Kuo, 2022): This method solves the generalized eigenvalue problem with $\hat{\Sigma}\_{b}^{-1} \hat{\Sigma}\_{f}$. As we demonstrate in our paper (Figure 2), directly inverting $\hat{\Sigma}\_{b}$ is numerically unstable when dimension $d$ is large. The reviewer might suggest stabilizing this by using a truncated pseudoinverse, similar to our PCA++. However, this reveals a deeper conceptual issue: this procedure is equivalent to projecting the foreground data $X\_{f}$ onto the principal subspace of the background data $X\_{b}$ and then performing PCA. Since the synthesized background $X\_{b}$ contains no signal $A$ by construction, its principal subspace is also signal-free. Projecting the foreground onto this signal-free subspace would annihilate the very signal we aim to recover. Therefore, a truncated cPCA++ would fundamentally fail.

Empirical Validation:

To investigate these different approaches empirically, we performed a new set of experiments comparing PCA++ against alternative methods proposed by the reviewer. We will add a full discussion of these results to the Appendix.

1. Baselines on synthesized data: Following the reviewer's suggestion, we evaluated methods that operate on synthesized foreground $X\_{f} = (X + X^{+})/2$ and background $X\_{b} = (X - X^{+})/2$ data. This includes standard cPCA and cPCA++ (based on a difference of covariances). This allows us to directly test the stability of the "subtract-then-decompose" approach versus our "decompose-from-cross-covariance" method.

2. Canonical Correlation Analysis (CCA): We also introduced CCA as a canonical and highly relevant baseline. We apply it by treating the paired data matrices, $X$ and $X^+$, as the two views. In our model, the shared signal is the sole source of population-level correlation between these views. Therefore, CCA, which finds directions of maximal correlation, is theoretically suited for recovering the signal subspace and serves as a strong benchmark.

We consider the following experiment setup:

• Experimental Setup: The experiment follows the $n=500$, $k=5$ fixed aspect ratio setting from Sec E.4. For cPCA, we set $\alpha=1$. This is the most principled choice as it yields an unbiased population estimator $\mathbb{E}[\hat{\Sigma}\_{f} - \hat{\Sigma}\_{b}] = AA^T + BB^{T}/2 + I\_d/2 - BB^{T}/2 - I\_d/2 = AA^{T}.$ We tested two scenarios: one with moderate background noise and one with strong background noise.

• Signal variances: [50, 25, 20, 15, 10]

• Moderate noise background: [100, 50, 40, 30, 20]

• Large noise background: [500, 400, 300, 200, 100]

Dear Reviewer uGvG,

Thank you for your thoughtful re-evaluation and for engaging so constructively with our work. Your feedback has been invaluable in helping us substantially improve the manuscript.

We recognize that your primary initial concern was the paper's novelty and positioning. We have worked diligently to address this, and the manuscript has been significantly enhanced as a result. The key improvements include:

1. Clarified novelty with extensive new baselines: We have thoroughly revised the paper to highlight our unique focus on robustness against strong, structured background noise. Based on your suggestions, we performed extensive new experiments comparing PCA++ against CCA and cPCA-style baselines, empirically demonstrating the superior stability of our GEV-based formulation.

2. Robustness beyond idealized assumptions: We have added new theoretical discussions and simulations to show that PCA++ remains effective even when key model assumptions like orthogonality, Gaussianity, and distinct eigenvalues are violated, confirming its practical utility.

3. Enhanced practical guidance: To make our work more complete, we have added a principled strategy for hyperparameter selection, a detailed computational scalability analysis, and a new Limitations and Future Work section.

Given that the original primary obstacles to acceptance have been fully addressed with new theoretical discussions and empirical results, we hope you'll agree that the manuscript in its current form is a strong, clear, and validated contribution.

We would be very grateful if you would consider the paper in its current, substantially improved state for a score more reflective of a clear accept.

Thank you again for your invaluable guidance throughout this process.

Best,

The Authors of Submission #26142

... the authors did not discuss the limitations of the work, for instance, with regards to the assumption or settings considered (Assumption 4.2, and 4.3), and how real-world data conforms to these assumptions.

A: We thank the reviewer for their thoughtful questions regarding the assumptions underlying our main theoretical results. We agree that discussing the scope and necessity of these assumptions is important. We address each point below and will add these clarifications to the camera-ready manuscript. (For a discussion of the paper's broader limitations, such as the linear model, please see the detailed response to Reviewer [9ppt].)

• On Assumption 4.1 (BBP detectability threshold): This assumption, $\lambda > \sqrt{c}$, is a classical and fundamental requirement in high-dimensional PCA for any spiked covariance model. As we note on lines 216-217, any signal spike weaker than this threshold becomes statistically indistinguishable from the noise bulk of the Marcenko-Pastur distribution. This is a fundamental limit of signal detection via PCA in this regime, not a limitation specific to our PCA++ method.

• On Assumption 4.3 (distinct eigenvalues): This assumption was made primarily for analytical convenience. With distinct eigenvalues, we can cleanly map each sample eigenvector to a population counterpart using existing results (Lemmas D.4, D.5). Without this assumption, if a set of eigenvalues were identical, we would instead estimate a subspace for those directions. Within that estimated subspace, it would be difficult to distinguish which basis vectors correspond to signal and which to background just by looking at $S_n$ alone, making it harder to explain how the uniformity constraint works on a per-direction basis.

The key insight is that even if the standard covariance $S\_n$ has degenerate subspaces (i.e., multiple identical eigenvalues mixing signal and background components), the contrastive covariance $S\_{n}^{+}$ resolves this ambiguity. Since $S\_{n}^{+}$ has asymptotically zero energy on pure background directions, the generalized eigenvalue problem can still correctly identify the signal subspace and separate it from the background.

To empirically validate this claim, we have run a new set of simulations for the setting in Sec E.4 (Figure 4) but with:

• Signal variances: [50, 50, 20, 15, 10]
• Background variances: [500, 500, 300, 50, 50]

The results below show that even with these degeneracies, the empirical subspace error for PCA++ continues to track our theoretical predictions, which were derived under the distinct spike assumption. This provides strong evidence that the assumption is a technical convenience rather than a practical necessity.

Fixed aspect ratio regime

Growing-spike regime

We will add a remark clarifying the role of these assumptions and include this new empirical validation in the appendix to strengthen our claims. Thank you again for pushing us to be more precise on these important details.

... I think it would be important to at least show how well the alignment-only PCA+ does on the data...

A: Thank you for your thoughtful feedback regarding the limitations of our real-world experiments and the importance of evaluating the uniformity constraint in contrastive PCA.

In response, we have included a comprehensive comparison across several methods, including standard PCA, alignment-only PCA+ (which lacks the uniformity constraint), PCA++, as well as additional baseline dimensionality reduction techniques (U-MAP, t-SNE, Robust PCA [Candes et al., 2011]). All methods were evaluated on the same single-cell RNA-seq data, focusing on their ability to recover cell-type groupings in both control and stimulated conditions.

The Adjusted Rand Index (ARI; lower is better in this context) for each method is summarized in the table below:

As shown, both PCA+ and PCA++ dramatically reduce the ARI compared to standard PCA and other popular methods, indicating much better mixing of the two conditions within cell types. Importantly, adding the uniformity constraint in PCA++ further improves the results over PCA+, especially in B cells and NK cells. These quantitative results highlight the utility of both the alignment and uniformity components. PCA++ achieves superior separation of invariant populations compared to alternative approaches. We will includ these expanded results and discussion in the revised manuscript.

I encourage authors to consider citing an important early work on contrastive PCA[1] and comparing the method with existing baselines.

A: We thank the reviewer for their suggestion to cite contrastive PCA (cPCA) [1] and for the prompt to compare with existing baselines. We will add this important citation and others from the cPCA literature to our related work section.

Regarding a direct comparison, it is important to note that our work and the cPCA method in [1] address fundamentally different problem settings:

• cPCA: This method requires two distinct datasets: a "foreground" dataset containing the signal of interest and an explicit "background" dataset. Its goal is to find variance in the foreground that is absent in the background.
• Our work (PCA++): We operate in the standard self-supervised learning setting, where we are given only paired positive views $(X, X^{+})$ of the data, where both views contain a mixture of the foreground signal and background context. Our method does not require access to a pure background dataset.

Because our setting does not provide the separate background dataset that cPCA requires, their method cannot be applied, which prevents a direct comparison.

For a more comprehensive discussion of the distinctions between our framework and the broader cPCA literature, we respectfully refer the reviewer to our detailed response to Reviewer [uGvG], where we elaborate on these points. Thank you again for bringing this important line of work to our attention.

I thank the authors for the additional experiments and the discussion on how cPCA methods relates to theirs. I highly encourage authors to include the discussion in the revised paper. It would make the position of the paper in the literature clear and enhance its readability.