Revisiting Residual Connections: Orthogonal Updates for Stable and Efficient Deep Networks

We appreciate the reviewer’s thoughtful feedback. Below, we address each concern, grouped by topic.

---

On ViT Baseline Performance and Robustness to Learning Rate

"The reported baseline (Table 2) for ViT-B on ImageNet-1K seems suspiciously low... Generalization performance is highly dependent on learning rates... A possibility of ruling out the learning rate dependence could be a quick logarithmic sweep..."

We thank the reviewer for this critical question. It touches upon two key aspects: the baseline's absolute performance and its robustness to learning rates.

• On the Baseline Performance and Training Paradigm: You are correct that our ViT-B baseline of 71.09% is lower than the ~85% reported by Google. This difference stems from a fundamental choice in training paradigm. The original ViT model's high performance relies on pre-training on the massive, non-public JFT-300M dataset. Our work, however, follows the more challenging and widely adopted academic paradigm of training from scratch on ImageNet-1k. As established by influential follow-up works like DeiT and T2T-ViT [1], this setting requires strong regularization (e.g., MixUp, CutMix, RandAugment) to prevent overfitting. Our experimental setup adheres to this modern, robust training recipe. Therefore, our baseline is a valid and fully converged result within this specific paradigm, not an undertrained model. This is further evidenced by the clear performance saturation shown in our accuracy curves.
• On Robustness to Learning Rate: We agree that the change in layer-wise gradient norms makes robustness to LR a crucial concern. To address this directly, we conducted an LR sweep on CIFAR–10, -100 for our ViT-S model. The results below demonstrate that our method's superiority is not an artifact of a single learning rate.

CIFAR-10 LR sweep results of Validation Accuracy@1 on ViT-S

CIFAR-100 LR sweep results of Validation Accuracy@1 on ViT-S

These results provide conclusive evidence that our method's superiority is not an artifact of a single, fortunate learning rate. The consistent performance advantage across a wide range of learning rates demonstrates the fundamental robustness of the orthogonal update mechanism. We will add this comprehensive sweep to the Appendix.

---

On the Interpretation of Table 7 and ResNetV2-50 Performance

"I am confused by Table 7: results show that the orthogonal residual update performs worse than the regular residual connections for all 3 initializations. Why is that? What initialization is used in Table 2?"

We thank the reviewer for pointing out this potential source of confusion. We apologize for not making the distinction clearer.

• Different Experimental Setups: The results in Table 7 are from a specific ablation study in Appendix C.1 designed solely to test initialization robustness. This experiment used ResNetV2-50 with our Global (Orthogonal-G) update and was run for a limited duration (60,000 steps, approximately 150 epochs) on CIFAR-100. In contrast, our main results in Table 2 use the superior Feature-wise (Orthogonal-F) update and report performance for fully trained models (200 epochs). As shown in Table 2, our primary Orthogonal-F method demonstrates clear performance gains on ResNetV2-18, -34, and -101.
• On ResNetV2-50 Performance: We acknowledge the performance gains on ResNetV2-50 are more modest. As we hypothesize in Section 4.2, we believe this is connected to architectural properties captured by our γ-ratio. The bottleneck design of ResNetV2-50 gives it a much higher γ-ratio (59.0) than ResNetV2-34 (14.75) . This suggests a nuanced interplay between representational width, architectural design, and our update's efficacy, which, as we state, "warrants further dedicated investigation."
• Initialization Method: For our main experiments in Table 2, we used standard initialization schemes appropriate for each architecture: Kaiming Uniform initialization for both ResNetV2 and ViTs, with batch normalization parameters initialized as described in Appendix C.1.

---

On Practical Runtime Overhead

"Can you provide a simple number on how much the runtime actually increased for the specific benchmarks?"

We thank the reviewer for this practical and important question. We agree that theoretical FLOPs do not always translate directly to runtime. To provide a direct answer, we have measured the practical training throughput (in images per second) for several of our key architectures on the hardware used in our experiments.

Throughput (sample images / sec) comparison table. higher is better.

As the results show, the practical throughput overhead of our primary Orthogonal-F update is modest. For ResNetV2 models, it ranges from ~3-13%, and for the high-capacity ViT-S and ViT-B model under torch.compile, the overhead is even lower at less than 2%. We believe this minor cost is a very favorable trade-off for the substantial gains in final accuracy and training stability that our method provides.

Furthermore, this small per-step overhead is often outweighed by improved convergence speed. As shown in Figure 2b, our method exhibits a significant time-to-accuracy advantage on large-scale tasks. For ViT-B on ImageNet-1k, our method reaches the baseline's final accuracy in ~40% less wall-clock time, demonstrating superior overall training efficiency.

We will add this detailed throughput analysis to the appendix.

---

Understanding the Distinction: Statistical vs. Geometric Orthogonalization

" ... As your method renders the signal from the main and residual branch uncorrelated for all frequency bins, do you think this could negatively affect trainability?"

The key distinction lies in how orthogonality is achieved. In [1], Frequency-Dependent Signal-Averaging (FDSA) emerges as a statistical byproduct - residual branches become less correlated at higher frequencies due to independent weight distributions and harmonic generation. Our method, however, implements geometric orthogonalization at each layer, ensuring $f_\perp(x_n)$ is explicitly perpendicular to $x_n$.

Why This Enhances Rather Than Impairs Trainability:

• Amplified Smoothing Effect: While [1] shows that residual connections provide "passive" FDSA benefits, our orthogonal updates actively maximize this effect. By explicitly removing the parallel component $f_\parallel$, we eliminate redundant signal modulation that could interfere with gradient flow, potentially achieving stronger blueshift mitigation than standard residual connections.
• Preserved Expressivity: Unlike the activation function modifications in [1] (e.g., Leaky ReLU slope adjustment) that reduce model expressivity to improve trainability, our method maintains the full computational capacity of each module. We only restructure how the module output is integrated into the residual stream, not the module's internal complexity.
• Deterministic Stability: Rather than relying on statistical decorrelation across frequencies, geometric orthogonalization provides deterministic norm preservation and directional diversity. Our experiments show this leads to more stable stream norms ($||x_n||^2$) and consistent rotational updates that guide representations toward new feature directions without magnitude explosion.
• Empirical Evidence: Our results demonstrate that this "complete orthogonalization" across frequency bins actually improves trainability, particularly in high-capacity architectures like Vision Transformers. The substantial performance gains (+4.3%p on ImageNet-1k) suggest that maximizing the FDSA principle through geometric means enhances rather than hinders the beneficial effects identified in [1].

In essence, our method can be viewed as taking the FDSA insight from [1] to its logical conclusion - if frequency-dependent decorrelation helps training, then explicit orthogonalization should help even more, which our empirical results confirm.

[1] Ringing ReLUs: Harmonic Distortion Analysis of Nonlinear Feedforward Networks (ICLR 2021)

---

Thank you again for your thoughtful engagement with our work. This discussion has been invaluable in helping us refine our arguments. We hope that our detailed response clarifies our contribution and resolves any remaining questions.

We sincerely thank the reviewer for their critical and detailed feedback. We believe some of the concerns may stem from a lack of clarity in our initial presentation, particularly regarding our motivation and the distinction from prior work. We are grateful for the opportunity to address these points.

---

On Motivation, Magnitude, and Novelty

"The motivation is not clear. Why the scaling operations caused by the parallel component is not necessary? The magnitude is also an important factor of updating... Are there any cited studies that support this viewpoint?"

We thank the reviewer for these insightful questions, which allow us to clarify the core principles of our work.

• Motivation and Magnitude: Regarding the lack of analysis on the original connection, our analysis in Appendix Figure 10, which shows the standard network learns to suppress the parallel component, directly addresses this by revealing a previously un-discussed inefficiency in the original design. Our core hypothesis is that high-capacity modules (e.g., attention, MLPs) are sub-optimally utilized if their capacity is spent on simple scaling operations (the parallel component, $f_\parallel$). Our method encourages the module to learn novel feature directions instead. We clarify that our method does not discard magnitude information from the stream $x_n$; it discards the part of the update that only scales the existing stream. In fact, Figure 3 (a, c) shows our method leads to a more stable stream norm, suggesting that removing the potentially conflicting parallel update aids in controlling signal magnitude.
• Empirical Justification: Our hypothesis is strongly supported by the learning dynamics of standard networks. As shown in Appendix Figure 10, when using a standard Linear Residual Update, the network itself learns to suppress the parallel component ($||f_\parallel||$ trends towards zero). Our method simply enforces this empirically discovered optimal behavior as a structural inductive bias from the start.
• Novelty: Our contribution is a novel approach to residual connections. Prior work on orthogonality focused on either
  • (a) regularizing network weights [2, 3] or
  • (b) modifying the skip-path itself with fixed transformations [4, 5].

Our approach is distinct: we preserve the standard identity skip-path and instead impose a dynamic, input-dependent geometric constraint on the additive update signal. The consistent empirical gains across diverse settings serve as strong validation for this new perspective.

---

Regarding the concern that the model's output space is not necessarily a Euclidean space

"The output space of the model is not necessarily a Euclidean space.”

We thank the reviewer for this sharp and important question. We completely agree that the feature space is best understood as a high-dimensional, curved manifold. This perspective is central to our method's motivation, and we are grateful for the opportunity to clarify this connection.

As the reviewer astutely notes, a key property of smooth manifolds is that they are locally Euclidean. Our method leverages precisely this property. The orthogonal decomposition $f(\sigma(x_n​))=f_\parallel ​+f_\perp$​ occurs in the tangent space $T_{x_n}​​M$ at the point $x_n$​ on the manifold, which is a vector space where Euclidean geometry locally applies.

Our update rule, $x_{n+1}​=x_n ​+f_{\perp}​(x_n​)$, can be rigorously interpreted as a first-order Taylor approximation of the exponential map,$exp_{x_n}$ $(f_\perp ​(x_{n}))$. The exponential map is the geometrically formal tool for moving along a curved manifold from a point $x_n$​ in a direction specified by a tangent vector $f_\perp​ (x_n​)$. The approximation is given by: $\text{exp}_{x_n}​​(v) \approx x_n ​+v$.

This linear approximation is highly accurate when the tangent vector $v=f_\perp​(x_n​)$ is small relative to the local curvature of the manifold. Our empirical results provide strong support for this condition. As shown in our paper's Figure 3 and Appendix Figure 10, the norm of our update vector, $||f_\perp​(x_n​)||$, consistently remains orders of magnitude smaller than the norm of the feature stream itself, $||x_n||$ (e.g., $\sim 10^1$ vs. $\sim 10^3$). This ensures the update is a small, local step on the manifold where the Euclidean approximation is valid.

Therefore, our method is not a naive application of Euclidean geometry to a global non-Euclidean space. Rather, it is a computationally efficient mechanism that leverages the local Euclidean nature of the learned manifold to guide feature exploration. This provides a novel inductive bias for navigating the feature space, and the strong empirical gains in our work validate this geometrically-motivated approach. We will add this clarifying discussion to the revised paper.

---

Relation to Prior Work

”The proposed orthogonal update rule can be rewritten as... similar with the skip connections of scaling in [1]... Table 1 in [1] shows that this update rule is worse."

This is an excellent point that highlights a crucial distinction. While the rewritten update rule is structurally reminiscent of scaling in [1], their purpose and effect are opposite, which explains the different outcomes. The distinction is fundamental:

• He et al. [1] : Uses a fixed hyperparameter (e.g., $\lambda_n$) for signal modulation. As they correctly showed, this can impede the identity gradient flow and harm performance.
• Our Method: Uses a dynamic function , $s_n(x_n, f(x_n))$, to compute a geometric projection. Its purpose is to enforce orthogonality, not to modulate the signal. Crucially, as shown in Eq. 5, this operation is explicitly designed to preserve the identity gradient path, which aligns with the core principle for stable training established by [1].

Therefore, our method is not a form of the detrimental scaling explored in [1], but a new geometric constraint compatible with the principles of deep residual learning.

---

On Experimental Scope and Paper Organization

"The organization of the paper is not well-structured... How does the proposed orthogonal residual update perform when applied to other commonly used modules and optimizers?"

We thank the reviewer for this feedback, and we will improve the paper's structure by adding a more explicit motivation in the Methods section. Regarding the experimental scope, our focus was strategic. To establish a general and foundational principle, we tested our hypothesis on the most fundamental and diverse building blocks of modern deep learning: the convolutional blocks of ResNetV2 and the Self-Attention/MLP modules of Vision Transformer. By demonstrating consistent gains with standard optimizers (SGD, AdamW) across these canonical architectures, our goal was to provide strong evidence for the universal applicability of the orthogonal update principle. We believe this focused study provides a clearer and stronger foundation for the community than a broader but less controlled survey of specialized modules.

---

[1] He, K., Zhang, X., Ren, S., & Sun, J. (2016). Identity mappings in deep residual networks. In ECCV 2016.

[2] Huang et al., Controllable orthogonalization in training dnns. CVPR 2020.

[3] Cisse et al., Parseval networks: Improving robustness to adversarial examples. PMLR 2017.

[4] Wang et al., Orthogonal and idempotent transformations for learning deep neural networks. arXiv:1707.05974.

[5] Lechner et al., Entangled residual mappings. arXiv:2206.01261.

---

Thank you again for your thoughtful engagement with our work. This discussion has been invaluable in helping us refine our arguments. We hope that our detailed response clarifies our contribution and resolves any remaining questions.

We sincerely thank the reviewer for their detailed feedback and insightful questions. We appreciate the opportunity to clarify these important aspects of our work. To be concise and address all points within the character limit, we will group our responses thematically.

---

On Effectiveness, Evidence, and Baselines (Weaknesses 1, 2, 3)

"The proposed method seemed to be less effective on the larger model... experiments did not provide strong evidence... reported results for ViTs on different datasets are relatively low..."

We thank the reviewer for these critical questions about our method's efficacy. We will address them cohesively.

• On Scalability and Effectiveness (W1): Our results demonstrate clear effectiveness on large-scale models. The +4.3%p absolute improvement for ViT-B on ImageNet-1k is a significant gain on a challenging benchmark, directly showing that our method's benefits scale effectively.
• On Evidence and Baselines (W2, W3): We agree the baseline performance requires context. Our primary goal was a rigorous 'paired comparison' to isolate the effect of the orthogonal update, not to set a new SOTA record. Our experiments follow the demanding "ImageNet-1k from scratch" paradigm, which, as established by works like DeiT, requires strong regularization (e.g., MixUp, CutMix) to prevent overfitting in Transformers. This explains why our baseline differs from models pre-trained on massive private datasets.
• On the Strength of Evidence: The evidence for our method's superiority is robust. As shown in Figure 2b, the performance gap between our method and the linear baseline is established early and consistently maintained or widened over 300 epochs. This stable, long-term advantage proves our method's efficacy is a genuine architectural effect, not an artifact of training duration. The consistency of these gains across diverse architectures (ResNetV2, ViT) and datasets provides a strong and cohesive body of evidence for our claims.

On Model Dynamics and Efficiency (Weaknesses 4, 5 & Questions 1, 2, 3)

"Could the authors provide more theoretical or intuitive explanations to Figure 3... Does it imply that the proposed orthogonal residual is trying to learn the linear residual...?" (W4, Q1)

This is an excellent question that probes the core dynamics of our method. The increasing cosine similarity does not imply our method is simply "learning to be linear." Instead, it reveals a fundamentally different and more controlled update mechanism compared to the standard linear update, a distinction best understood by examining Appendix Figure 10.

• Linear Update's Dynamic: "Explosive" Reinforcement. In a standard Linear Update, the module can directly reinforce the existing feature stream $x_n$. As observed in Figure 10, this can lead to moments of "explosive" updates around the transition point, where the norm of the parallel component $||f_\parallel||^2$ grows significantly. This suggests the standard method sometimes aggressively scales existing representations.
• Orthogonal Update's Dynamic: Controlled "Compensation" via Leakage. In contrast, our Orthogonal Update structurally prevents this direct, potentially unstable reinforcement. The module learns to produce a large parallel component in its full output, but this is explicitly discarded from the update vector. We hypothesize that the module then relies on the small, controlled $\epsilon$-leakage (mathematically defined in Eq. 9) to perform the necessary, subtle scaling adjustments to the stream. This leakage acts as a controlled "compensation" mechanism, allowing for stable, fine-tuned modulation rather than the direct, sometimes explosive, reinforcement seen in the linear case.

Therefore, the increasing cosine similarity is a sign of a more sophisticated adaptation. The module learns to handle the main information flow (via the parallel component of its full output), while our method ensures the actual update remains a controlled injection of novel information, using leakage for fine-tuning. This is a key finding about how networks adapt to and leverage geometric constraints for more stable learning.

---

"Why does the stability constant have a relatively large effect on the performance of the proposed method?" (Q2)

The constant ϵ ensures numerical stability but also plays a subtle optimization role. As shown in Appendix B.1 (Eq. 9), a non-zero $\epsilon$ allows a small, controlled amount of the parallel component's gradient information to "leak" through. This can be beneficial as it prevents the optimization from stalling in regions where a purely orthogonal update might have near-zero gradient with respect to certain loss components, thus aiding learning stability. Our ablation in Figure 5 shows performance is robust across several orders of magnitude, with $\epsilon=10^{-6}$ providing the best stability in our experiments.

---

"How does the normalization affect the proposed method... the norm of the proposed residual update is very distinct..." (Q3)

The reviewer's observation is correct and points to a key benefit: our method provides intrinsic geometric stability. The distinct norm behavior is a direct result of our geometric constraint. When the update norm is much smaller than the stream norm ($|| f_\perp (x_n) || \ll ||x_n ||$ ), which is empirically validated in our figures , the update $x_{n+1} = x_n + f_\perp (x_n)$ acts as an approximate rotation. This dynamic causes the representation to effectively "walk on a hypersphere," preserving the stream's magnitude ( $|| x_{n+1} || ^ 2 \approx || x_n || ^2$ ) while changing its direction. This provides an inherent mechanism for norm control, which is particularly beneficial for architectures like Transformer that rely heavily on explicit normalization layers for stable training.

---

"The authors have mentioned faster convergence... Could the authors provide more efficiency comparisons in the paper?" (W5)

The paper provides two clear efficiency comparisons.

1. Computational Cost: Section 3.4 and Table 1 present a detailed FLOPs analysis, demonstrating that the theoretical overhead of our method is marginal compared to the main module's complexity (e.g., Attention or MLP blocks).
2. Practical Efficiency: More importantly, Figure 2b provides a direct wall-clock time-to-accuracy comparison. It shows that our method not only achieves a higher final accuracy but also reaches any given accuracy target in significantly less time. This demonstrates superior overall training efficiency, where the small computational cost per step is heavily outweighed by faster convergence.

---

Formatting Concerns

"... some figure legends could be made bigger and clearer."

Thank you for the suggestion. We will ensure all figure legends are improved for clarity and readability in the camera-ready version.

---

Thank you again for your thoughtful engagement with our work. This discussion has been invaluable in helping us refine our arguments. We hope that our detailed response clarifies our contribution and resolves any remaining questions.

We sincerely thank the reviewer for their valuable feedback and insightful questions. The comments help us clarify the core contributions of our work and the context of our experimental results. We are happy to provide detailed responses and additional analysis below.

---

On the Paper's Contribution and Positioning

"It is not clear whether the main claim of this study is to “propose a practical method” or to "deepen the basic understanding of shortcut paths"... please clarify the main point of this paper."

We thank the reviewer for this excellent question, as it allows us to clarify the core positioning of our work. Our primary contribution is a fundamental study that provides a new understanding of residual connections, with our proposed method serving as a practical proof-of-concept that validates our hypothesis.

Our fundamental hypothesis is that the parallel component of the standard residual update ($f_\parallel$​) can be redundant or even detrimental, and that encouraging novel feature directions via Orthogonal Residual Update leads to more efficient representation learning. A deeper analysis, supported by Appendix Figure 10, reveals that the standard Linear Residual Update consistently learns to suppress the parallel component, with its norm approaching zero during training. This strongly supports our hypothesis that this component may be unnecessary.

In contrast, the dynamics of our Orthogonal Residual Update are more complex and intriguing than a simple removal of a redundant component. The increasing cosine similarity seen in Figure 3 is not a monolithic phenomenon. Instead, it results from at least three different observed behaviors across layers:

• Both orthogonal and parallel components of the full module output increase (e.g., MLP block 0).
• The orthogonal component stabilizes or decreases, while the parallel component is maintained or grows (e.g., Attention blocks 2, 3, 4).
• Both components decrease, but the orthogonal component does so more slowly (e.g., Attention block 5).

This complex behavior does not contradict our hypothesis. Instead, it reveals that our method, by acting as a strong inductive bias, does not entirely eliminate the parallel component from the module's full output, but rather re-purposes its role within the optimization dynamics. We believe this non-trivial, counter-intuitive result about how networks adapt to geometric constraints is a key contribution to the fundamental understanding of shortcut paths.

Finally, the practical aspect of our work serves as the direct validation of this hypothesis. Our strictly controlled experiments (Table 2, Figure 2) show that this fundamental change leads to consistent improvements in performance and stability. Crucially, as our new quantitative analysis confirms, this performance gain stems from the model learning demonstrably richer and more diverse feature representations. This provides the essential empirical proof for our primary, more fundamental contribution. We will clarify this positioning in the revised manuscript.

---

On Feature Richness and Cosine Similarity Dynamics

"The discussion of the experimental results in Figure 3 alone is not convincing enough... Do the extraction of the vertical components added by the proposed method really provide “richer features”?"

We agree that our claim of "richer features" requires direct, quantitative evidence. As attaching new visualizations is not possible during the rebuttal period, we have conducted a comprehensive supplementary analysis on the feature representations learned by both the Linear and our Orthogonal ViT-B models on the full ImageNet-1K validation set. We extracted the 768-dimensional CLS token features from the final transformer block (before LayerNorm) of our fully trained models, using the 288-epoch checkpoint from the complete 300-epoch training run. This ensures our analysis reflects the final, converged state of the representations.

The results, summarized below, consistently demonstrate that our Orthogonal Residual update produces richer and more diverse representations.

$\text{Effective Rank (Participation Ratio)} = (\Sigma_{i} p_{i}^2​)^{−1}$ where $p_i$​ are the normalized singular values.

$\text{Spectral Entropy} = - \Sigma p_{i} * log(p_{i})$

Key Findings:

1. Higher Global Diversity: We measured the Effective Rank of the feature covariance matrix. A higher rank indicates that features span a higher-dimensional space and are less collapsed. Our method achieves a 4.7% higher effective rank, providing strong evidence that it encourages the model to learn a richer global representation.
2. Fundamentally Different Structure: We used Centered Kernel Alignment (CKA) to measure structural similarity. A score of 1.0 would imply identical representations. The resulting score of 0.546 demonstrates that the features learned by our method are structurally different from those of the standard update, supporting our claim that the orthogonal update is a meaningful inductive bias.
3. Enhanced Diversity with Superior Stability: Critically, our method achieves greater representational diversity while simultaneously enhancing numerical stability. The feature standard deviation is 52.5% lower, indicating that the features are more tightly clustered and well-behaved. This is not a trade-off, but a synergistic improvement: our orthogonal updates encourage the model to explore a wider range of feature directions without resorting to large-magnitude, potentially unstable feature values that can hinder optimization.

These complementary metrics robustly support our conclusion that the Orthogonal Residual update leads to richer, more effective feature representations. We will include this detailed analysis in the appendix of the revised paper.

---

On ViT Baseline Performance and Training Convergence

"it is possible that the Top-1 Accuracy is low, especially in the Linear settings of ViT-S and ViT-B, and that the training has not converged sufficiently."

We thank the reviewer for this critical question. The baseline performance is a direct and expected consequence of the specific, challenging training paradigm we chose to ensure a fair comparison.

• The "ImageNet from Scratch" Paradigm: The original ViT paper demonstrated that without massive pre-training (e.g., on JFT-300M), ViTs underperform CNNs on ImageNet-1k. To address this, influential follow-up works like DeiT and T2T-ViT established a new training recipe for this from scratch setting, which requires strong regularization (e.g., MixUp, CutMix) to prevent overfitting. Our experimental setup adheres to this modern, robust recipe.
• A Fair Comparison at Practical Convergence: Our 300-epoch training schedule ensures a robust comparison at a point of practical convergence. As shown in Figure 2(b), the performance advantage of our method is not a short-term artifact. The gap is established early and is consistently maintained or widened throughout the vast majority of the training process. The rate of improvement for both models slows significantly in the final 100 epochs, indicating that while absolute performance might marginally increase, the stable, long-term relative gap between the methods is not in doubt. This makes our comparison a valid and reliable measure of the orthogonal update's efficacy.

Therefore, we are confident that our results represent a scientifically valid comparison between two methods at a point of performance saturation, and the reported improvements robustly reflect our method's efficacy.

[1] Yuan, L., et al. Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet. ICCV 2021.

---

Thank you again for your thoughtful engagement with our work. This discussion has been invaluable in helping us refine our arguments. We hope that our detailed response clarifies our contribution and resolves any remaining questions.